We are scaling our infrastructure and processing thousands of concurrent tasks daily. Our core engineering question is whether Airflow is still dominating data pipelines at this extreme volume, or if the scheduler becomes a massive bottleneck. We have faced database locks in our metadata layer using LocalExecutor. Would moving to Celery or Kubernetes executors solve this bottleneck completely, or should we look at other options?
3 answers
Airflow can easily scale to millions of tasks if configured with the right distributed executor. The common performance bottlenecks usually stem from poor database tuning or using a basic executor like the LocalExecutor which runs tasks on a single machine. By migrating your pipelines to the KubernetesExecutor or CeleryExecutor, you distribute the heavy workloads across isolated pods or workers. Furthermore, optimizing settings like the file processing interval and scaling up your metadata database instances will alleviate the locks and ensure your scheduler runs perfectly under heavy loads.
I am considering this migration for our team too. Does migrating to the KubernetesExecutor significantly increase the task latency due to pod spinning times, or is it negligible for long-running ETL processes?
Transitioning to a distributed executor transformed our delivery. Our database locks completely disappeared once we isolated task executions into dedicated nodes.
That matches our experience perfectly. Tuning the pool sizes alongside the executor migration is critical because it prevents your workers from overwhelming external databases during concurrent runs.
The pod creation overhead is usually around a few seconds, which is completely negligible if your processing tasks run for several minutes or hours. However, if your data pipeline consists of hundreds of micro-tasks that execute in seconds, you might find the CeleryExecutor better since it uses workers that are already up and waiting.