Many of our senior architects are pushing to move all our ETL pipelines from MapReduce to Spark. Is there still any valid use case where traditional MapReduce is superior? I’m concerned about the memory requirements of Spark and whether it’s as robust for massive 100TB+ batch jobs that don't need real-time results.
3 answers
Spark has largely overtaken MapReduce because it processes data in-memory, making it up to 100x faster for iterative algorithms. However, MapReduce is still incredibly robust for "embarrassingly large" batch jobs where data doesn't fit in RAM. Since MapReduce is disk-based, it can handle massive spills to disk more gracefully than Spark, which can sometimes throw OOM (Out of Memory) errors if not tuned perfectly. If you have a simple one-pass ETL job on 500TB of data and you don't care if it takes 6 hours instead of 1, MapReduce is often cheaper and more stable on commodity hardware.
Have you looked into the cost of ownership? Spark usually requires higher-end instances with more RAM, whereas MapReduce is happy on cheaper, storage-heavy nodes. Is the speed worth the cloud bill?
Spark is the future for developer productivity. Writing 10 lines of Scala/Python in Spark is much better than 200 lines of Java for a simple WordCount in MapReduce.
I agree with Nancy. The time we save on development and debugging with Spark's high-level APIs far outweighs the slightly higher infrastructure costs in the long run.
Gary, that is our main debate. In our AWS EMR environment, the Spark clusters are costing us roughly 40% more per hour. We’ve decided on a hybrid approach: we use Spark for our interactive analytics and machine learning workloads where time-to-insight is critical, but we’ve kept our massive, weekly data-cleansing batch jobs on MapReduce. It’s slower, but the reliability on cheap spot instances is hard to beat for those specific non-urgent tasks.