I have a FastAPI endpoint that accepts large CSV uploads for processing. I want to return a 202 Accepted status immediately and run the data cleaning and validation in the background. Is the built-in BackgroundTasks feature sufficient for tasks taking 5-10 minutes, or should I integrate Celery and Redis?
3 answers
For tasks taking up to 10 minutes, FastAPI's BackgroundTasks is generally not recommended. It runs within the same process as your web server. If your server restarts or crashes, you lose the task state. For Data Science pipelines, I highly suggest Celery with RabbitMQ or Redis. This decouples the heavy processing from the API workers. It also allows you to scale your "worker" nodes independently from your API nodes, which is vital when you start doing heavy Pandas or NumPy operations that consume significant RAM.
If I move to Celery, how do I report the progress of the data cleaning back to the user? Can FastAPI easily query the Celery task status to show a progress bar?
I agree on Celery. BackgroundTasks are really only for quick things like sending a confirmation email after a user signs up.
Exactly, Sandra. Using the right tool for the job prevents your main API from becoming sluggish.
Yes, you can! You create a GET endpoint like /status/{task_id}. FastAPI then checks the Celery AsyncResult. If you store custom state in Celery, you can return the exact percentage of the CSV processed, giving your users a much better experience than just a spinning wheel.