I am looking to move from batch processing to a real-time streaming architecture. In the world of Data Engineering, is it better to use a managed service like AWS Glue, or should I build a custom solution using Apache Kafka and Flink? I need to handle roughly 10TB of event data daily and want to ensure my schema remains consistent across all downstream consumers.
3 answers
Transitioning to real-time Data Engineering is a major architectural shift. If your team is small, I strongly recommend a managed service like AWS Glue or GCP Dataflow to reduce operational overhead. However, for 10TB a day, the costs can spiral quickly. A custom Kafka and Flink setup offers much more granular control over backpressure and state management. The "Gold Standard" here is using a Schema Registry to prevent downstream breakage. In my experience, a well-tuned Flink job can handle millions of events per second with sub-second latency, but you’ll need a dedicated DevOps resource to manage the cluster health and scaling policies effectively.
Are you planning to use a Lakehouse architecture like Delta Lake or Apache Iceberg for the storage layer of this Data Engineering project?
I’d suggest looking into "dbt" for the transformation layer. It has become an industry standard in modern Data Engineering for managing SQL-based logic.
I totally agree with Megan! Using dbt within your Data Engineering workflow allows you to treat your data transformations like code, complete with version control and automated testing.
That is a great question, Justin! I’m actually leaning toward Delta Lake. I want to ensure that our Data Engineering pipeline supports ACID transactions so we don't end up with partial writes during a cluster failure. Using an open-table format seems like the best way to keep our data accessible for both BI tools and machine learning models without having to constantly move files around between different storage buckets and databases.