I'm deploying a generative AI model and need to stream results back to the client word-by-word. I am using FastAPI's StreamingResponse class. Are there specific configurations for Uvicorn or Gunicorn that I should tune to prevent memory leaks when multiple users are requesting long-running inference tasks simultaneously?
3 answers
Streaming large outputs in FastAPI requires careful management of the event loop. Use an asynchronous generator to yield data from your model. If your model is CPU-bound (which most Deep Learning models are), don't run the inference directly in the async route. Instead, use run_in_executor or a task queue. For Uvicorn, increase the timeout_keep_alive and consider using h11 or httptools as the loop/parser to maximize throughput. Also, ensure you are not buffering the entire response in memory before yielding; yield chunks as they are generated.
How do you handle client-side disconnects? If a user closes the tab, does FastAPI automatically stop the heavy inference task, or do we need to manually check for a disconnected state?
I use WebSockets in FastAPI for this instead of StreamingResponse. It feels more robust for bi-directional AI interactions.
WebSockets are great for interactivity, especially if the user needs to interrupt the model mid-stream!
You need to monitor the Request.is_disconnected() flag within your generator loop. If it returns true, you should break the loop and stop the model inference. This is crucial for Deep Learning apps because it prevents wasting expensive GPU cycles on requests that no one is watching anymore.