We are setting up a small production Hadoop cluster with 8 nodes for internal analytics. I see a lot of documentation about NameNode HA, but is setting up YARN ResourceManager High Availability overkill for a cluster this small? What happens to running MapReduce jobs if the ResourceManager fails and we haven't configured the Zookeeper-based failover?
3 answers
Even for a small cluster, YARN HA is highly recommended if your business relies on those analytics. Without it, the ResourceManager (RM) is a single point of failure. If the RM goes down, all currently running applications fail immediately, and no new jobs can be submitted until you manually restart it. By using Zookeeper, you can have an Active and a Standby RM. The state of the applications is saved in a 'State Store' (usually in HDFS or Zookeeper), so when the Standby takes over, it can actually resume the work without losing everything.
Does the 'State Store' you mentioned include the logs of the finished jobs, or just the metadata for the ones that were currently in the queue when the crash happened?
For an 8-node cluster, the overhead of Zookeeper might be annoying, but it's better than getting paged at 3 AM because a master node service crashed.
I second that. We started without HA on a small 5-node test rig and regretted it the first time we had a simple hardware failure. It's worth the setup time.
Thomas, the RM State Store primarily keeps track of the application attempts, delegation tokens, and container statuses for active jobs. Finished job logs are actually handled by the JobHistoryServer, which is a separate daemon. So, for HA, you are mostly protecting your "in-flight" work. If you have long-running Spark streaming jobs or 6-hour MapReduce batches, YARN HA isn't overkill; it's a necessity to avoid massive rework and wasted compute hours.