We’re migrating from an old Informatica ETL setup to a modern ELT flow using dbt and Snowflake. I’m concerned that loading raw data first will cause our storage costs to skyrocket or slow down our final reports. Is ELT truly the standard now, or are we just shifting the "transformation" bottleneck from the server to the warehouse?
3 answers
ELT is the standard because cloud warehouses like Snowflake are designed to scale compute and storage separately. By doing transformations inside Snowflake (ELT), you leverage its massive parallel processing power. In our migration last year, we actually found it faster than our old ETL server because we weren't limited by a single machine's CPU. As for costs, storage is cheap; it's the "compute" you have to watch. If you use "Incremental Models" in dbt, you only process new data, which keeps your Snowflake credits under control. It's a huge shift in mindset but much more scalable.
Melissa, how do you handle "data quality" in an ELT world? If you load the raw data first, doesn't that mean you're potentially filling your warehouse with garbage before you even have a chance to clean it?
We saved $20k a month switching to ELT. The ability to use SQL for everything via dbt made our whole team much more productive.
Danielle, that ROI is hard to argue with. The productivity gain from having everyone speak the same "SQL language" is a massive hidden benefit of ELT.
Steven, that's what the "Bronze" layer is for! You land the raw data in a dedicated schema, then you run dbt tests immediately. If the data fails a quality check (like a null value in a primary key), you stop the pipeline before it hits your "Silver" or "Gold" layers. It's actually better because you have the raw data available for debugging if something goes wrong, whereas in traditional ETL, that raw source might be gone once the transformation fails.