We have a bunch of NVIDIA GPUs and want to use kubernetes to manage our Deep Learning experiments. However, we're worried about one job hogging all the GPU memory and crashing other services. Are there specific schedulers or plugins that help with GPU slicing and fair-share scheduling?
3 answers
Running AI workloads on kubernetes is very common now thanks to the NVIDIA Device Plugin. To prevent one job from hogging resources, you should use "GPU Time-Slicing" or "Multi-Instance GPU" (MIG) if you have A100/H100 cards. This allows you to carve a single physical GPU into multiple smaller virtual GPUs. Also, consider using Volcano, which is a batch scheduling engine specifically built for high-performance computing on kubernetes. It handles job queuing and priority much better than the default scheduler, ensuring your training jobs don't starve your web services of resources.
Do you have a plan for data locality? Moving terabytes of training data into the kubernetes pods can be a major bottleneck if your storage isn't optimized.
Use Kubeflow. It’s a dedicated platform for machine learning on kubernetes that handles everything from notebooks to distributed training and model serving.
Kubeflow is definitely the way to go for serious teams. Megan’s advice on the NVIDIA plugin is also a prerequisite for any kubernetes AI project.
We are using an S3-compatible local storage, but I’m worried about the I/O speed. Chris, do you think we should use a high-performance filesystem like Lustre, or can we get away with standard NVMe drives mapped directly to the kubernetes nodes? We need the data to be available to the containers as fast as possible to keep the GPUs at 100% utilization.