I need to re-run a data pipeline for the entire month of January to fix a calculation error. I’m worried that if I just clear the old tasks, it will overwhelm my database connections or interfere with the current daily runs. What is the safest way to perform a backfill in Apache Airflow while ensuring that the current production schedules still have priority?
3 answers
The safest way to backfill is using the Airflow CLI command airflow dags backfill. Unlike clearing tasks in the UI, the CLI command allows you to specify a date range and runs independently of the scheduler's normal cycle. To protect your production runs, you should use "Pools." Assign your DAG tasks to a specific pool with a limited number of slots. This ensures that even if you trigger a month’s worth of backfills, they can only occupy, say, 5 slots at a time, leaving the rest of your worker capacity for the "real-time" daily tasks that need to run concurrently.
Have you set the depends_on_past parameter to True for this DAG? If you are backfilling, the order of execution might matter significantly for your calculation fix.
You can also use the max_active_runs parameter in your DAG definition. Setting this to 1 during the backfill will prevent the scheduler from trying to do too much at once.
Good tip, Larry. Lowering max_active_runs is a quick way to throttle the DAG without having to mess with the global pool settings if you’re in a hurry.
Walter, that’s a crucial reminder. Since this fix involves a cumulative sum, each day depends on the previous day’s results being correct. If I don't set depends_on_past=True, Airflow might try to run all 31 days of January in parallel, which would result in the same calculation errors we had before. I will combine the CLI backfill with the Pool strategy Kimberly mentioned to ensure we stay within our API rate limits while maintaining the correct sequential order for the data.