I’m torn between using vLLM and Ollama for our internal RAG system. Ollama is so easy to set up, but I’ve heard that vLLM is much faster for multi-user scenarios. Since we expect about 50 concurrent users, which one should we invest our engineering time into for the long term?
3 answers
For 50 concurrent users, vLLM is the clear winner. Ollama is fantastic for local development and single-user prototyping, but it uses a static memory allocation that causes requests to queue up under load. In contrast, this engine uses PagedAttention and continuous batching, which allows it to process dozens of requests simultaneously on a single GPU. In my benchmarks from late 2023, vLLM provided nearly 10x the throughput of Ollama once we moved past 5 concurrent users. The learning curve is steeper, but it’s the only way to meet strict production SLAs.
Do you have a dedicated DevOps person to manage the server, or are you looking for a "one-click" deployment solution?
If you go with the more advanced engine, make sure to use a vector database like Pinecone to keep your retrieval speeds high.
Great point, Susan. Even the fastest inference engine like vLLM won't help if your RAG retrieval step takes three seconds. Both parts of the pipeline need to be optimized for low latency.
We have a small team, Daniel. While we value ease of use, we can't sacrifice the latency that our users expect. If vLLM requires more configuration but delivers a better UX, we are willing to put in the 20-30 hours of setup time.