We are planning to run our high-volume analytics workflows on an existing enterprise infrastructure framework. Instead of going with a standalone implementation, we want to look at running a Spark cluster managed through Hadoop YARN. Could someone explain how resource requests are negotiated here? How do the master roles change when we shift from standalone cluster managers over to YARN architecture?
3 answers
When executing workloads over YARN, the traditional standalone master process is completely replaced by YARN’s Resource Manager. When you submit a job via your terminal, the framework allocates a container to spin up an Application Master. In client mode, your driver program runs locally on your workstation, while the Application Master merely schedules structural container resources from YARN NodeManagers. In cluster mode, your driver process runs entirely inside the remote Application Master container itself, making it highly fault-tolerant and ideal for heavy production runs.
Does shifting your architecture over to YARN require manual cluster tracking, or are you utilizing automated logging tools to capture runtime metrics?
You will need to pass the --master yarn flag explicitly when calling your execution scripts to route the resource allocations through the YARN manager daemon.
Peter is absolutely right. Forgetting that argument defaults the job execution to local context, which defeats the purpose of utilizing a distributed compute engine.
YARN abstracts a lot of the operational tracking natively. You can query application timelines directly via the Resource Manager interface. However, for deep debugging, enabling Spark’s history server with an underlying HDFS log repository is vital to inspect application behaviors after jobs complete.