We are moving our generative AI models from standalone instances to a shared Kubernetes cluster. I’m struggling with GPU fragmentation and underutilization. Is anyone using NVIDIA's Multi-Instance GPU (MIG) or specialized schedulers like KubeRay to handle the bursty nature of inference? How do you ensure low-latency responses when multiple pods are competing for the same hardware resources?
3 answers
We’ve had great success using the NVIDIA GPU Operator alongside MIG. The biggest trick was not just enabling MIG, but setting up specific node pools with "Time-Slicing" for smaller dev models while reserving full A100/H100 slices for production inference. We also integrated the KEDA autoscaler to trigger scaling based on the number of pending requests in our model queue rather than just CPU/Memory metrics. This setup reduced our cold-start times by 30% and significantly improved our hardware ROI.
Jessica, that KEDA approach sounds promising. Are you seeing any significant scheduling overhead or "noisy neighbor" issues when running different model versions on the same physical GPU?
We’ve found that using "PriorityClasses" is essential. It ensures our critical user-facing AI models always pre-empt lower-priority data cleaning jobs when GPU capacity gets tight.
Exactly. Without PriorityClasses, your production traffic is at the mercy of whatever job started first. It's a fundamental part of the 2025 AI-on-K8s playbook.
Robert, we actually use Pod Anti-Affinity rules to ensure that high-priority inference pods never share a GPU with background batch processing tasks. For noisy neighbors, we rely on the GPU Operator’s device monitoring to kill any process that leaks memory or exceeds its allocated power limit, which keeps the node stable.