I'm trying to containerize our inference stack. Does anyone have a reliable Dockerfile for that includes the necessary CUDA drivers and optimization flags? I am particularly concerned about how to expose the OpenAI-compatible server port while ensuring that the container can access all available GPU resources on the host machine.
3 answers
For an enterprise-grade deployment, I recommend starting with the official NVIDIA PyTorch image as your base. You need to ensure you install the library with the specific CUDA version that matches your host drivers. Inside the Dockerfile, make sure to set the environment variables for your model directory so the container doesn't re-download weights on every restart. When running the container, use the '--gpus all' flag and map port 8000. This setup allows you to utilize the standard vLLM.entrypoints.openai.api_server as the main process, which is highly optimized for production.
Have you run into any issues with shared memory limits when running large models inside Docker?
Using a multi-stage build can really help keep the final image size down, which is crucial for fast scaling.
Good point, Sean. Reducing the image size from 12GB to 5GB made our deployment times on AWS much faster during peak traffic periods.
Yes, that is a common pitfall. You should always set the '--shm-size' flag to at least 1g or higher when running multi-GPU setups. If the shared memory is too low, the NCCL backend used for communication between GPUs will often crash with a cryptic error. Increasing the shared memory at the container level solved all our stability issues when running the Llama-2 70B model in our Kubernetes clusters.