We are finding that purely semantic vector search in Haystack sometimes misses specific product SKUs or technical jargon. Is it possible to build a "Hybrid Search" engine that uses both BM25 and Dense Retrieval simultaneously? How do we handle the ranking of results from both sources to ensure the most relevant documentation appears first for the user?
3 answers
Hybrid search is definitely the way to go. Combining the ElasticsearchDocumentStore with neural retrievers in Haystack provides the best of both worlds for accuracy and speed.
Implementing hybrid search is exactly where Haystack shines. You can create a pipeline with two parallel branches: one for the InMemoryBM25Retriever and another for a DensePassageRetriever. You then pipe both into a JoinNode or, even better, a Ranker component. In early 2024, I used the SentenceTransformersRanker to re-score the combined results. This ensures that even if the vector search is a bit "fuzzy" on the SKU, the keyword search catches it, and the ranker puts the most contextually relevant one at the top. It drastically improved our search precision for specialized industrial parts.
Does using a Ranker component add a noticeable delay to the query response time? We are aiming for sub-second latency for our customer-facing AI search engine, and I'm worried that adding a re-ranking step after retrieval might push us over that threshold.
Scott, it does add a bit of latency, usually around 100-200ms depending on the model size. However, you can optimize this by only re-ranking the top 10-20 results rather than the whole set. For an AI search engine, the gain in quality usually outweighs the tiny speed hit. If you use a lightweight Cross-Encoder, you can keep the total round-trip under 500ms easily, which still feels instantaneous to the end-user.
I totally agree. It’s the standard setup for any serious search application. Hybrid search makes the system much more robust against varied user query styles.