I am setting up a production environment for a high-traffic AI application. I’ve decided to use vLLM because of its PagedAttention algorithm, but I’m struggling to find the optimal tensor-parallel size for my four A100 GPUs. Should I prioritize lower latency or maximum throughput when configuring the serving parameters for such a large model?
3 answers
To maximize efficiency with vLLM, you should set your --tensor-parallel-size to 4 to match your GPU count. This allows the 70B model to be sharded across all units, which is necessary since the weights alone exceed the memory of a single A100. I also recommend enabling --enable-prefix-caching if your application involves repetitive system prompts. In my tests back in 2023, this reduced redundant KV cache computation significantly. For maximum throughput, keep your --max-num-seqs high, but monitor your GPU memory utilization closely to avoid out-of-memory errors during peak bursts.
Are you planning to use a specific quantization method like AWQ or FP8 to further reduce the VRAM footprint on those A100s?
I found that switching from standard Hugging Face Transformers to the vLLM engine cut our inference costs by nearly 60% due to the better batching.
I totally agree, Susan! The continuous batching feature in vLLM is what makes it the gold standard for production. It effectively eliminates the idle time between requests that usually plagues other serving frameworks.
I am leaning towards FP8 because it offers the best balance between speed and precision loss. Does vLLM support native FP8 kernels for the newer Blackwell architecture yet, or should I stick to AWQ for now to ensure stability in my production cluster?