We’ve recently migrated to a Medallion Architecture using Bronze, Silver, and Gold layers. However, we're struggling with data quality issues leaking into the Gold layer. How are you all handling automated data profiling and observability to catch schema drift before it hits our BI dashboards?
3 answers
Implementing 'Great Expectations' or 'dbt tests' at the transition between the Bronze and Silver layers is non-negotiable. You should be validating schemas and checking for null values or outliers immediately after ingestion. In the Medallion setup, the Silver layer should represent your 'cleaned' source of truth. If you wait until the Gold layer to check for quality, the lineage becomes too complex to untangle. I also recommend using tools like Monte Carlo for automated lineage tracking, which helps identify exactly where a pipeline broke and which downstream assets are impacted.
Are you currently using any specific orchestration tools like Airflow or Prefect to trigger these quality checks, or are they integrated into your Spark jobs?
Focus on circuit breakers. If the Silver layer fails a DQ check, the Gold update shouldn't run. This keeps your BI reports accurate, even if they are slightly delayed.
Absolutely. A "late" report is always better than a "wrong" report in a production environment. Automated alerts via Slack or PagerDuty are essential here.
We are using Managed Airflow on AWS. We want to find a way to halt the DAG execution if a critical data quality test fails in the Silver processing stage to protect our Gold tables.