I am trying to build a streaming pipeline using the Medallion Architecture (Bronze, Silver, Gold). Should I use Delta Live Tables (DLT) or stick to standard Structured Streaming? I am specifically worried about handling late-arriving data and schema evolution without breaking the downstream Gold tables.
3 answers
Delta Live Tables (DLT) is the way to go if you want to simplify the operational complexity of the Medallion Architecture. It handles the underlying infrastructure and checkpointing for you. For late-arriving data, you can use the "expectations" feature to define data quality rules. If data doesn't meet the criteria, it can be dropped or flagged for review. Schema evolution is also much cleaner in DLT; you can set it to automatically evolve the schema or fail the pipeline if a breaking change is detected, ensuring your Gold tables remain clean.
Are you dealing with high-volume streaming data where sub-second latency is required, or is a minute-level refresh rate acceptable for your business use case?
Use DLT for the "declarative" approach. It allows you to focus on the SQL or Python logic of the data transformation rather than managing the cluster lifecycle.
Exactly, Rachel. The fact that DLT handles the DAG generation automatically makes it much easier to onboard new developers who might not be Spark experts yet.
Brian, a minute-level refresh is perfectly fine for our reporting needs. We aren't doing high-frequency trading, just real-time inventory monitoring. My concern is mostly about how DLT handles retries when a transient network error occurs during the Silver to Gold transformation. Is there a built-in retry logic that doesn't require us to write custom exception handlers?