I am concerned about the overhead of an orchestration layer. If I use the Haystack framework to build a multi-hop RAG system, will the pipeline execution add significant latency compared to a custom Python script? We are targeting a sub-2-second response time for a knowledge base of 50,000 documents. Looking for real-world benchmarks from the community.
3 answers
The latency in the Haystack framework is actually quite negligible when compared to the time taken by the LLM inference itself. In our benchmarks, the framework overhead was less than 10ms per pipeline run. Most of your delay will come from the Retriever (vector DB latency) or the Generator (LLM API). What this framework provides is a way to optimize those specific parts. For example, you can use the CachingRetriever to skip redundant searches or implement parallel component execution. Since it's built on top of standard Python asyncio in the newer versions, it handles concurrent requests very well. If you are hitting 50k documents, your bottleneck will be the vector database index type, not the framework orchestration logic.
Which vector store are you pairing with the Haystack framework? I've found that the connection protocol (gRPC vs HTTP) usually has a bigger impact on the overall RAG latency than the framework code itself.
We achieve 1.5s response times with a complex Haystack framework setup. The key is to optimize your document store and use a fast generator like Groq or a local vLLM instance.
Exactly, Cynthia. The framework is just the glue. If the glue is thin (which it is here), the speed depends entirely on the components you choose to stick together.
We are currently using Qdrant with gRPC. You're right, David, the framework isn't the slow part. After testing, I found that the 'Re-ranker' node we added was adding 500ms. The framework actually helped us identify this because of its built-in OpenTelemetry support. We could see exactly where every millisecond was being spent in the trace, which led us to optimize the cross-encoder model we were using.