A core selling point of the Hadoop Distributed File System (HDFS) is its robust fault tolerance. Could someone detail the mechanism of data replication? What is the default replication factor, and where are those replicas typically placed (e.g., on which DataNodes)? More importantly, how does Rack Awareness come into play to prevent data loss during a massive failure event, like a complete network switch or power failure for an entire rack in the Cloud Technology cluster?
3 answers
HDFS achieves robust fault tolerance through automatic data replication at the block level. The default replication factor is 3. When a client writes a file, the file is broken into blocks, and these blocks are replicated according to a specific placement policy to maximize reliability and minimize network traffic. The standard policy places: 1) the first replica on the local DataNode (if the client is a DataNode), 2) the second replica on a DataNode in a different rack, and 3) the third replica on a different DataNode within the same rack as the second. Rack Awareness is the crucial network topology knowledge that enables this distribution, ensuring that a single point of failure (like an entire rack going down due to power or network issues) does not lead to data loss, as at least one replica is guaranteed to reside on a physically separate rack within the Cloud Technology cluster.
That distribution strategy based on Rack Awareness makes perfect sense for resilience. If a DataNode fails, the NameNode initiates re-replication to restore the factor of 3, but how quickly can a massive HDFS cluster recover from a multiple-node failure simultaneously, especially if the workload from the Big Data processing job (e.g., a huge MapReduce or Spark job) is also saturating the network? Is there a performance trade-off between the high fault tolerance and the overall cluster throughput during recovery operations?
HDFS ensures fault tolerance by replicating data blocks (default factor is 3) across different DataNodes and using Rack Awareness to place replicas on separate physical racks, guaranteeing data availability even if an entire rack fails.
The use of commodity hardware in Cloud Technology means failures are frequent. This block-level data replication is the reason HDFS is so robust and cost-effective for petabyte-scale Big Data storage, as it makes hardware reliability a software problem instead of a hardware one.