I'm currently working on a pipeline that needs to process hundreds of individual files daily. I’ve been using a for-loop inside my DAG to create tasks dynamically, but I’ve noticed the Apache Airflow webserver is becoming incredibly sluggish. Is there a more efficient way to handle dynamic tasks at scale without compromising the UI performance or overloading the scheduler's parsing time?
3 answers
The issue you are describing is quite common when using traditional Python loops for task generation because the scheduler has to parse that logic every few seconds. To fix this, you should switch to Dynamic Task Mapping, which was introduced in version 2.3. Instead of creating 500 individual task objects in your code, you define one task and use the .expand() method. This allows Airflow to create task instances at runtime based on the length of your input list. This significantly reduces the size of the DAG structure the webserver needs to render, making your UI much more responsive.
Have you checked if your top-level code includes any heavy database queries or external API calls? Sometimes the lag isn't just from the number of tasks but from what the scheduler has to execute just to read the file. Are you using the TaskFlow API for these dynamic tasks?
You should definitely look into the expand() and partial() functions. They are designed specifically to handle high-volume task instances without bloating the metadata database.
I agree with Patrick. I switched our ETL to mapping last year and the DAG load time dropped from 12 seconds to under 1 second. It's a total game changer for scale.
That's a great point, Gregory. If she is doing something like os.listdir() or a SQL query at the top level of the DAG file, it executes every time the scheduler heartbeats. Combining Dynamic Task Mapping with a dedicated task that fetches the file list first is the way to go. This keeps the "heavy lifting" inside an actual task instance rather than in the DAG parsing logic itself, which should solve the UI performance issues immediately.