I keep hearing that "Data Locality" is the primary reason why Hadoop is more efficient than traditional storage systems. Can someone explain the technical process of how YARN and HDFS coordinate to move the code to the data rather than moving the data across the network? I want to understand the impact this has on network bandwidth in 2025.
3 answers
Data Locality is fundamental to the "Shared Nothing" architecture of Hadoop. When you submit a job, the YARN Resource Manager consults the NameNode to find the physical location (IP address) of the DataNodes holding the required blocks. YARN then attempts to schedule the TaskAttempt on the same node where the data resides—this is called "Node Local." If that node is busy, it tries to schedule it on a node in the same rack, known as "Rack Local." By avoiding the "Data Motion" across the Top-of-Rack switches, you save massive amounts of network bandwidth, which is often the most expensive and slowest resource in a Big Data cluster.
In a cloud environment like AWS or Azure, where compute and storage are often separated (like using EMR with S3), is the concept of Data Locality still relevant, or does high-speed networking make this architectural principle obsolete for modern cloud-native developers?
Data Locality is the only reason MapReduce scales to petabytes. Without it, your network switches would literally melt under the pressure of the data shuffle phase.
Nancy is spot on. Even with Spark, we still rely on the underlying HDFS locality to ensure that our initial RDD partitions are loaded as efficiently as possible from the disk.
Robert, that’s the million-dollar question. In the cloud, we often trade "Data Locality" for "Elasticity." While S3 doesn't offer true HDFS-style locality, we use local NVMe caching and high-throughput 100Gbps links to bridge the gap. However, for extreme low-latency processing, a co-located HDFS cluster still beats a remote object store every single time in terms of raw throughput.