I have a trained Random Forest model and I need to serve it to a web application. I'm torn between using Flask, FastAPI, or a dedicated tool like BentoML or Seldon. What are the performance trade-offs for a data science team that doesn't have a massive DevOps budget but needs to handle around 100 requests per second with low latency?
3 answers
For 100 requests per second, FastAPI is significantly better than Flask because it supports asynchronous requests natively, which is crucial for handling multiple concurrent hits without blocking the event loop. However, if you want something "production-ready" out of the box, BentoML is excellent because it handles the containerization (Docker) and model versioning for you. It packages your Scikit-Learn model into a "Bento" that can be deployed anywhere. If you are on a tight budget, a simple FastAPI wrapper deployed on an AWS Lambda or a small ECS cluster is usually the most cost-effective path while maintaining sub-50ms latency for most tabular models.
Are you planning to handle the preprocessing (like scaling or one-hot encoding) inside the API or as part of a Scikit-Learn Pipeline object? I’ve seen people hit major "training-serving skew" because their API code handled missing values differently than their training notebook.
If you already use a cloud provider, look at SageMaker or Vertex AI. It might be more expensive than a DIY FastAPI, but it saves hours of infrastructure management.
True, Laura, but the costs can spiral. Robert, if you go the FastAPI route, make sure to use Gunicorn with Uvicorn workers to ensure your server can actually scale across CPU cores.
Christopher, that’s a vital point! To avoid that skew, I always save the entire Pipeline object using joblib. When the FastAPI app loads the model, it’s loading the scaler and the encoder too. One thing to watch out for with FastAPI is the Pydantic validation overhead. If your input JSON is massive, the validation can actually take longer than the model prediction itself. For Robert's 100 RPS requirement, using ujson instead of the standard library can shave off a few milliseconds.