Our team is debating between Text Generation Inference and vLLM for our upcoming project. Since we are dealing with high traffic, we need to know which one offers better beam search performance and better utilization of NVIDIA A100 GPUs. Does anyone have benchmarks comparing the two for real-time applications?
3 answers
We conducted extensive internal benchmarking on A100s and found that while TGI is very stable and has great features like continuous batching, the other framework outperformed it in raw requests per second. The secret is the PagedAttention algorithm which manages the memory tokens in a way that allows for much larger batch sizes. We reached about 15x higher throughput on Llama-2-70b. However, be aware that the memory management is quite aggressive, so you need to monitor your GPU temperatures and power draw closely when you are pushing it to the absolute limit in a live cluster.
Interesting results! Did you notice any significant difference in the "Time to First Token" (TTFT) metrics between the two frameworks during your high-load tests?
We moved to it last month and the integration with OpenAI-compatible APIs made the transition for our frontend team seamless.
That API compatibility is a huge selling point; it meant we didn't have to rewrite our entire client library to start seeing the performance gains.
The TTFT was actually slightly better on TGI for single requests, but as soon as we added concurrency, the specialized memory handling of the newer framework took the lead. It keeps the latency consistent even when the queue starts building up, which is vital for UX.