We are migrating our raw storage to a Data Lakehouse using Apache Iceberg. I keep seeing the "Medallion Architecture" (Bronze, Silver, Gold) mentioned as the gold standard. For those of you who have implemented it, how do you strictly define the transition from Silver to Gold? Is the Gold layer strictly for aggregated metrics, or can it include flattened dimensional tables for PowerBI?
3 answers
In our Lakehouse setup, we treat the Silver Layer as the "Source of Truth"—this is where data is cleaned, filtered, and joined, but still granular. The transition to the Gold Layer is strictly for consumption-ready data. We use Gold for two things: highly aggregated fact tables for executive dashboards and flattened "Wide Tables" optimized for tools like PowerBI. The key is that users should never query Silver. By keeping the Gold layer as a set of specific data products, you can apply strict Data Governance and quality checks. If a column name changes in the source, you only have to fix the Silver-to-Gold logic, keeping your end-user reports stable and reliable.
Are you using Table Formats like Delta Lake or Hudi alongside Iceberg to manage the ACID transactions between those different layers?
We found that using dbt to manage the transitions between Bronze, Silver, and Gold made our lineage so much easier to track during compliance audits.
Sarah is right. Visualizing the Data Lineage is crucial for knowing exactly which Silver table is feeding into your Gold executive summary.
David, Iceberg is usually enough for the ACID part. Alice, regarding your definition of Gold, we actually separate it by "User Personas." Our Data Scientists get access to the Silver layer for training their Machine Learning models because they need the raw features. Our Business Analysts only ever see the Gold layer. This "dual-access" strategy prevents the Gold layer from becoming a cluttered mess while still empowering the power users who need the raw, un-aggregated details.