I am configuring a high-availability (HA) cluster with two NameNodes. I know I need Zookeeper to manage the failover (ZKFC). My question is about the Quorum size. Is 3 nodes enough, or should I go with 5 or 7 for a cluster with 100 DataNodes? Also, does the physical location of these Zookeeper nodes impact the failover speed?
3 answers
For a cluster of 100 DataNodes, a 3-node Zookeeper Quorum is usually sufficient, as Zookeeper's performance actually decreases slightly as you add more nodes (due to the overhead of reaching consensus). You always want an odd number to avoid "Split-Brain" scenarios. A 5-node quorum is only necessary if you want the ability to lose 2 nodes simultaneously while maintaining the cluster. Regarding location: yes, latency is critical. Zookeeper nodes should be on the same network segment as the NameNodes. If the network between ZK and the NameNode is slow, the ZK Failover Controller (ZKFC) might think the active NameNode is dead just because a "heartbeat" packet was delayed, leading to a "flapping" HA state.
If I use 3 nodes and one goes down for maintenance, I only have 2 left. Does Zookeeper still function with just 2 nodes, or does the entire NameNode HA fail because a "majority" (2 out of 3) is still technically present?
Don't put Zookeeper on the same physical disks as your DataNode data. Zookeeper needs fast, low-latency disk I/O for its logs to ensure the quorum stays healthy.
Great tip, Elizabeth. Using dedicated SSDs for Zookeeper logs has drastically reduced our "session timeout" errors in the cluster.
Thomas, if you have a 3-node cluster and 1 is down, the remaining 2 can still form a quorum because 2 is a majority of 3. However, if you lose one more, the whole thing stops. This is why many people prefer 5 nodes; it allows you to take one node down for maintenance and still have a "safety buffer" of one more failure before the quorum is lost.