We are currently redesigning our backend and the debate has come up again: why Airflow is still dominating data pipelines when there are so many event-driven alternatives? We deal with heavy ETL loads and need something that handles complex dependencies without breaking. For those working in high-growth environments, does Airflow still hold the crown for reliability, or are you looking elsewhere for your orchestration needs?
3 answers
In our organization, we stuck with Airflow for our data pipelines because of its robust handling of "backfilling." When you're dealing with historical data and need to re-run logic for the past six months, Airflow’s execution date logic is flawless. Most newer tools focus on "now," but data engineering often requires looking backward. We manage about 500 TB of data moving through various stages daily, and the way Airflow manages task dependencies via DAGs ensures that we never have data integrity issues due to race conditions.
Melissa, how do you deal with the "Python dependency hell" when different data pipelines require different library versions? It feels like the worker nodes get cluttered very quickly. Are you using Docker for every single task, or is there a more efficient way to manage the environments without increasing the latency of the task startup?
For us, the main reason it dominates is the sheer number of integrations. Whether it's Snowflake, Databricks, or even legacy SQL servers, there's always a reliable Hook or Operator available.
Well said, Christina. The extensibility is the real winner. We’ve even written custom sensors to trigger our data pipelines based on internal API states, which was surprisingly easy to implement given the well-documented base classes in the Airflow source code.
Gregory, we actually use the PythonVirtualenvOperator for smaller variations, but for heavy lifting, we definitely lean on the KubernetesPodOperator. It ensures each task in our data pipelines runs in a completely isolated container. While there is a slight startup latency for the pod to pull the image, the trade-off in environment stability and the ability to use different languages (like Spark or even C++ binaries) within the same DAG is well worth those extra few seconds