Most of the documentation I see for DeepSpeed focuses on training and ZeRO stages. However, I’m interested in the "DeepSpeed-Inference" component. Has anyone used it to serve Llama-2 or Llama-3 models? I want to know if the kernel fusion and transformer kernels actually provide a 2x speedup as claimed, or if it's mostly marketing hype for enterprise users.
3 answers
The inference acceleration in DeepSpeed is quite legitimate, especially for latency-sensitive applications. It works by replacing the standard Hugging Face implementation of layers with highly optimized CUDA kernels. This includes "kernel fusion," where multiple operations (like layer norm and activation) are fused into a single GPU kernel to reduce memory trips. In my experience with Llama models, I saw about a 1.5x to 1.8x increase in tokens per second. The main catch is that it doesn't support every single model architecture out of the box, so you have to check the compatibility list first.
Are you running this on a single GPU or a multi-GPU setup with tensor parallelism enabled through the DeepSpeed runtime?
I’ve used it for BERT-based models in production. The latency reduction was significant, dropping our p99 response times by nearly 40% without any loss in accuracy.
That’s a massive improvement, Larry. It goes to show that even for "smaller" models, those optimized CUDA kernels make a world of difference in a high-traffic environment.
Kevin, the performance gains are actually most noticeable in multi-GPU setups. DeepSpeed-Inference manages the tensor parallelism automatically, which is much faster than the manual 'device_map' approach in standard Transformers. It significantly reduces the inter-GPU communication lag during the auto-regressive decoding phase, which is usually the biggest bottleneck for LLMs.