My organization is moving away from Hadoop and we want to run our Spark jobs on EKS (Kubernetes). Are there significant performance penalties when using K8s as the cluster manager? Also, how do we handle dynamic resource allocation effectively without the traditional NodeManagers we had in YARN?
3 answers
Running Spark on Kubernetes (K8s) has become very mature in the last two years. The main advantage is "Cloud Native" isolation and faster scaling. You don't have a performance penalty in terms of CPU/RAM, but you do need to be careful with the networking and shuffle data. For dynamic resource allocation, Spark 3.0+ added support for the "External Shuffle Service" on K8s, but a more modern approach is using "Shuffle Tracking," which allows executors to be removed only when they aren't holding active shuffle blocks. Using the Spark Operator for K8s makes managing these jobs much easier than raw kubectl commands.
Susan, I've heard that the "Shuffle Tracking" feature can be a bit flaky under heavy load. Have you compared it to using an external storage layer like S3 for shuffle data? Some people say that moving shuffle to the cloud is the only way to truly scale on K8s without local disk bottlenecks.
Start with the Spark Operator. It handles the lifecycle of your driver and executors much like YARN did, but within the native Kubernetes ecosystem and its security policies.
I agree. The operator approach is much more "Agile." It allows us to treat our Spark clusters as code, which fits perfectly into our existing CI/CD pipelines for our data science models.
Kevin, using S3 for shuffle is an option, but it introduces high latency. A better middle ground is using an "External Shuffle Service" pod on each K8s node. This way, the shuffle data stays local to the node's NVMe drive even if the Spark executor pod is terminated, giving you the speed of local disk with the flexibility of K8s's ephemeral pod lifecycle.