Our team is moving toward a micro-DAG architecture in our cloud environment to keep things modular. However, we are struggling with downstream triggers. When one DAG finishes, we need three others to start. Is it better to use the TriggerDagRunOperator or should we rely on Sensors to monitor the state of the upstream Apache Airflow tasks? What are the pros and cons for cloud costs?
3 answers
In a cloud environment like MWAA or Astronomer, the choice between TriggerDagRunOperator and ExternalTaskSensor often comes down to resource consumption. If you use a standard Sensor, it stays "running" and occupies a worker slot, which can get expensive and lead to worker starvation. The best modern approach is to use "Deferrable Operators" or "Datasets." Datasets (introduced in 2.4) allow you to trigger a DAG based on a URI update, which is completely event-driven. If you must use a sensor, ensure it is in reschedule mode so it doesn't hold onto a slot while waiting.
Have you considered using the TriggerDagRunOperator with a specific wait_for_completion=False setting to keep it asynchronous? It seems like you want to avoid the "long-polling" cost of sensors entirely.
For cloud-native setups, I highly recommend using the new Dataset-based scheduling. It removes the need for explicit operators and makes the dependencies feel much more natural.
Justin is right; Datasets are the future for cross-DAG logic. It simplifies the UI view and makes the relationship between producers and consumers much easier to audit for the DevOps team.
Have you considered using the TriggerDagRunOperator with a specific wait_for_completion=False setting to keep it asynchronous? It seems like you want to avoid the "long-polling" cost of sensors entirely.