We are deploying vLLM on a GKE cluster with multiple nodes. However, we keep hitting NCCL timeout errors during the model initialization phase. Could this be related to the shared memory size or the network interconnect between the nodes? We really need this to be stable for our enterprise-grade AI deployment.
3 answers
NCCL errors in a containerized vLLM environment are almost always related to the --shm-size. By default, Docker and K8s pods have a very small shared memory limit. You need to increase this to at least 16GB or use the host's IPC namespace. Additionally, check if your nodes are connected via a high-speed interconnect like NVLink. If you are going across nodes, ensure that your NCCL_IB_DISABLE and NCCL_P2P_DISABLE variables are set correctly based on your network topology. I spent two weeks debugging this in late 2024 before realizing it was just a resource limit issue.
Have you tried running a basic NCCL-test container on your nodes to verify the point-to-point bandwidth before launching vLLM?
Make sure you are using the official Docker image from the vLLM repository, as it comes with the optimized CUDA libraries pre-installed.
Exactly, Gregory. Trying to build vLLM from source in a custom container often leads to version mismatches with the NCCL binaries which are very hard to track down manually.
That’s a smart diagnostic step, Jason. I found that if the underlay network isn't optimized for RDMA, the vLLM engine will wait indefinitely for weight synchronization across the tensor-parallel shards, leading to the timeouts Rachel is seeing.