I’ve been using mlflow models serve for local testing, but I need to move our sentiment analysis model to production. What is the standard way to deploy MLflow models into a Kubernetes cluster? Should I be wrapping the model in a custom FastAPI app, or is the built-in Docker container strategy robust enough for handling thousands of requests per second?
3 answers
For production-grade high availability, you should use the mlflow models build-docker command to create a standardized image. This image is optimized and includes all necessary dependencies. Once you have the image, you can deploy it using a Kubernetes Deployment with a LoadBalancer. To handle thousands of requests per second, you'll need to configure Horizontal Pod Autoscaling (HPA) based on CPU or request latency. While the built-in server is solid, some enterprises prefer wrapping the model logic in FastAPI if they need custom middleware for authentication or complex pre-processing that isn't captured in the model's signature.
Is there a way to integrate MLflow serving with a service mesh like Istio to manage the traffic splitting for A/B testing?
We use Seldon Core to wrap our models. It has a native integration that makes the deployment onto our cluster almost entirely automated.
Seldon is a great choice! It adds that extra layer of monitoring and logging that the basic serve command lacks, especially for production monitoring.
Yes, Steven, that is a very common pattern. You can deploy two different versions of your MLflow model as separate K8s services and then use Istio VirtualServices to route a percentage of traffic to each. This gives you a very powerful MLOps setup where you can verify the performance of a 'Challenger' model against the 'Champion' in real-time without any downtime for your end users.