I'm studying for a Big Data certification and I'm stuck on the concept of Rack Awareness. I understand it's about putting replicas on different racks, but how does the NameNode actually "know" the physical topology of the data center? Is there a script I need to write, or does Hadoop detect the network switches automatically to prevent data loss during a rack-level power failure?
3 answers
Hadoop does not automatically detect your network topology. You have to provide a "topology script" (usually a Python or Bash script) and configure the net.topology.script.file.name property in core-site.xml. When a DataNode registers with the NameNode, the NameNode runs this script, passing the DataNode's IP address. The script returns a rack path like /rack-01. By default, Hadoop's replication policy places the first replica on the local node, the second on a different rack, and the third on a different node within that same second rack to balance performance and safety.
If you don't configure this script, doesn't Hadoop just assume every single node is on one giant default rack called /default-rack, thus defeating the purpose?
Rack awareness is also vital for "Data Locality." It helps the TaskTracker or YARN NodeManager prioritize tasks on nodes where the data actually lives, saving bandwidth.
Exactly! By knowing the rack, YARN can pick a node on the same rack if the local node is busy, which is much faster than pulling data across the main backbone switch.
Yes, Richard, that is exactly what happens. If you leave it at default, a single switch failure could take out all three replicas of a block if they happened to be placed on nodes connected to that switch. Configuring the script is mandatory for any production-grade cluster. You can even test it using the command hdfs dfsadmin -printTopology to see if your nodes are correctly mapped to their respective physical locations in the data center.