We are designing an advanced machine learning workflow where tasks need to pass data matrices between steps. I often see people arguing about why Airflow is still dominating data pipelines if its built-in XCom system is only intended for small metadata elements. Should we rely heavily on XCom for custom pipeline communication, or is it better to completely decouple state sharing by using external object storage?
3 answers
The core rule of Airflow orchestration is to keep your control plane separate from your data plane. XCom was intentionally built to pass small operational details like execution timestamps, status flags, or file paths across tasks, storing them securely in the relational metadata database. If you attempt to pass large data structures or pandas dataframes through standard XComs, your database storage will balloon rapidly, severely hurting cluster performance. For massive datasets, you should write the task output to an object store like Amazon S3 and pass only the file URI via XCom.
That separation makes complete sense for performance. Has anyone tried the custom XCom backends feature introduced in newer versions to automate this object storage process?
We learned the hard way that dumping large data vectors directly into XCom strains the system metadata storage. Moving to an S3 bucket path approach solved our slowness.
Absolutely, keeping the database light is essential. Passing file pointers keeps your task execution clean and ensures the UI remains snappy when inspecting historical data pipeline runs.
Yes, implementing a custom XCom backend is brilliant because it abstracts the entire process. You can configure it to automatically serialize dataframes straight to S3 or GCS behind the scenes, allowing your python functions to return objects naturally without manual cloud client coding.