Our data team is currently overwhelmed with maintaining complex Glue jobs and managing schema drift in our ETL pipelines. I’ve been reading a lot about the Zero-ETL approach being promoted by AWS and Snowflake. Does it actually eliminate the need for data engineering, or is it just another marketing buzzword? How are you all handling the integration of transactional data into your warehouses without the high latency of batch processing?
3 answers
Zero-ETL is definitely a powerful trend, but it isn't a "magic wand" that removes the need for data engineers. It essentially provides a native, low-latency replication between your operational database and your Cloud Data Warehouse. While this eliminates the "E" and "L" for specific supported sources, you still need engineers for the "T"—the transformation. Raw data is rarely ready for BI tools. In my experience, Zero-ETL is fantastic for getting data into your environment quickly, but you still need a solid dbt (data build tool) layer to clean up the messy JSONs and normalize schemas. It shifts your focus from infrastructure maintenance to data modeling, which is where the real business value actually lies.
Have you considered if your current Change Data Capture (CDC) setup is the actual bottleneck before moving to an entirely new Zero-ETL framework?
I think of it as "Managed ETL." It saves our team from the "janitor work" of fixing broken connectors, letting us focus on building better Data Models for our analysts.
I agree with Jessica. Reducing the manual "plumbing" of pipelines is the only way for a small data engineering team to scale with the company's growth.
Steven, that is a fair question. Mark, the "Zero-ETL" hype often masks the fact that it only works within specific ecosystems. If you are using a multi-cloud strategy, you’ll still need a robust Data Orchestration tool like Airflow or Dagster. We found that while AWS Aurora to Redshift Zero-ETL saved us time, we still spent 40% of our week fixing downstream breaks because the "automatic" integration didn't account for our specific business logic transformations.