I'm building a generative AI app where the image generation takes about 30 seconds. If I use a standard endpoint, the client often times out. Should I be using WebSockets, or is there a way to use BackgroundTasks in FastAPI to handle this? I want the user to get a "task started" message and then get the result later. What's the industry standard here?
3 answers
For 30-second tasks, the industry standard is to use a task queue like Celery with Redis. However, for simpler needs, FastAPI’s built-in BackgroundTasks works well. You would return a "Task ID" to the client immediately with a 202 Accepted status. The client then polls a separate status endpoint. If you want a more "real-time" feel, WebSockets are the way to go. FastAPI makes WebSockets incredibly easy to implement, allowing you to stream progress updates (like "50% rendered") back to the user. This keeps the connection alive and provides a much better UX than a spinning wheel that eventually leads to a 504 Gateway Timeout.
If I use BackgroundTasks, does that run in the same memory space as the API? I'm worried about the AI model consuming all the RAM and crashing the server.
StreamingResponse is another cool option in FastAPI. You can literally stream the tokens from an LLM to the client as they are generated, just like ChatGPT.
I agree, the StreamingResponse is perfect for LLMs. It makes the application feel much faster because the user starts seeing results almost instantly rather than waiting for the whole block.
You hit the nail on the head, Richard. BackgroundTasks does share memory with your main app. For heavy AI models, this is risky. That’s why for production-grade deep learning, we usually offload the task to a separate worker process or a serverless function. This way, your API remains responsive even if a heavy inference job is saturating the CPU or GPU on a different worker node.