We are currently struggling to balance our need for real-time analytics with the consistency of our nightly batch jobs. Has anyone successfully implemented a Lambda Architecture using Apache Spark and Hadoop? I am curious about how you handle the serving layer to ensure users don't see duplicate data when the batch layer finally syncs with the speed layer.
3 answers
Implementing Lambda successfully requires a very robust serving layer, usually something like Apache Druid or even a NoSQL solution like Cassandra. The trick to avoiding duplicates is to use a unique transaction ID for every record. When the batch layer runs, it essentially "overwrites" the speculative data in the speed layer with the "source of truth." In my experience at a fintech firm, we used Spark Streaming for the speed layer and MapReduce for the batch. We ensured that the query logic in the serving layer always prioritized the batch-calculated values once they were timestamped after the speed layer's window.
That sounds like a lot of overhead. Have you looked into Kappa Architecture instead? It simplifies things by treating everything as a stream, which removes the need to maintain two separate codebases for batch and speed. Why stick with the complexity of Lambda if your stream processor can handle historical data?
We actually use Delta Lake to solve this. It provides ACID transactions over your big data, so the "merging" of batch and stream happens almost automatically at the storage level.
Jennifer is right. Delta Lake has really changed the game. Sarah, if you're just starting your implementation, look into Lakehouse architectures; it might save you from the "dual pipeline" headache Emily mentioned earlier.
Michael, the reason we stay with Lambda is for "re-processability." If a bug is found in the streaming logic, the batch layer serves as a permanent, immutable backup that we can re-run to fix the historical state. Kappa is great, but for high-compliance industries like ours, having that separate batch "gold standard" is a regulatory requirement. It’s definitely more work, but the data integrity is much higher when you have those two independent paths validating each other.