Our organization is finally planning to upgrade our legacy Hadoop 2.x clusters to version 3.3. I’ve read about Erasure Coding and support for GPU isolation. From a developer's perspective, what are the most significant changes in the YARN architecture that we should be aware of to optimize our distributed applications?
3 answers
The jump to Hadoop 3.x is massive. The most significant feature for storage efficiency is Erasure Coding, which reduces the storage overhead from 3x (with replication) to about 1.5x, effectively doubling your usable capacity on the same hardware. For developers, YARN now supports "Opportunistic Containers," which improves cluster utilization by running low-priority tasks in idle slots. Furthermore, the native support for GPU resources in YARN is a game-changer for those of us running Machine Learning and Deep Learning workloads directly on the cluster. You also get multiple NameNode support for better high availability, which removes the single point of failure more robustly than before.
Erasure Coding sounds great for saving space, but doesn't it significantly increase the CPU and network overhead during the reconstruction of missing data blocks compared to simple 3-way replication?
Don't forget that Hadoop 3.x finally moved to Java 8/11 minimum, which allows us to use modern lambda expressions and streams in our MapReduce code!
Absolutely, Linda! Moving away from Java 7 was long overdue and makes the developer experience so much more pleasant when writing custom UDFs for Hive or Pig.
Kevin, you've identified the main trade-off. Erasure Coding is "CPU-heavy" but "Disk-light." It is best used for "Cold Data"—information that you need to keep for compliance but don't query every minute. For "Hot Data" that requires maximum read performance, sticking with standard HDFS replication is usually the better strategy to avoid the reconstruction latency.