I am developing a Flask application that needs to process large CSV files after a user uploads them. Since the processing takes several minutes, the HTTP request times out. I want to move this task to a background thread so the user gets an immediate "Success" message while the server works in the back. Is the Python threading module safe to use with Flask's application context, or should I be looking at dedicated task queues like Celery or Redis?
3 answers
Using the standard threading.Thread module is the simplest way to start. You can define a function for your task and trigger it inside your route using thread = threading.Thread(target=your_function, args=(data,)). However, the biggest hurdle is the Flask Application Context. If your background task needs to access the database via SQLAlchemy or use current_app, it will fail because the new thread doesn't have the request context. You must manually use with app.app_context(): inside the thread function to give it access to your app's resources. While fine for small projects, it doesn't scale well if the server restarts, as all active threads will be killed instantly.
That context manager tip is huge, but what happens if the background thread encounters an error? Since it's decoupled from the main request, how do you track if the CSV processing actually finished or if it crashed halfway through without the user ever knowing?
For anything serious, skip threading and use Celery with Redis. It acts as a separate worker process that persists even if your web server reboots, ensuring tasks aren't lost.
I agree with Richard. I started with threading like Christopher, but quickly realized that Celery's ability to retry failed tasks and handle horizontal scaling made it much more reliable for our enterprise-grade Python deployments.
Jessica, that's exactly why most production systems move away from simple threads. To handle errors properly, you'd usually log the output to a database table. A better approach for monitoring is using a status flag in your DB. The thread updates the status to 'Running' when it starts and 'Completed' or 'Failed' when it ends. This allows you to create a separate AJAX endpoint that the frontend polls to show a progress bar to the user.