We are currently orchestrating our dbt Core models using Apache Airflow (MWAA). Management is considering moving to dbt Cloud for the integrated IDE and documentation features. Is the per-seat pricing actually justified for a team of 15 engineers, or are we better off building a custom CI/CD wrapper?
3 answers
For a team of 15, the cost of dbt Cloud adds up quickly. The primary value-add of the Cloud version is the "Slim CI" and the managed environment that handles the state of your models. However, you can replicate 90% of this in Airflow using the "Cosmos" operator by Astronomer. It automatically parses your dbt project and turns it into an Airflow DAG. This gives you the best of both worlds: the power of dbt's SQL transformations and the enterprise-grade scheduling of Airflow. If your team is already comfortable with Python and Git, the "per-seat" tax of dbt Cloud might not offer enough incremental value to justify the switch.
Does your team rely heavily on the dbt Cloud IDE for development, or are they mostly working locally with VS Code and just using the cloud for production runs?
If you have non-technical analysts who need to run and document models, the dbt Cloud UI is much more accessible. But for pure engineers, the Airflow route is more flexible.
Exactly. It's a "User Experience" vs. "Customization" trade-off. For a data engineering heavy team, stick with Core and Airflow.
Michael, we are 100% VS Code users. To answer your question, the IDE isn't the draw for us; it’s the "Artifact Management." But as Susan mentioned, we’ve started using an S3 bucket to store our manifest.json files between Airflow runs. This allows us to run "incremental" tests on only the modified models in our CI pipeline. It took about a week to set up the bash script, but it saved us thousands of dollars a year in licensing fees while keeping our workflow identical to the Cloud offering.