Standard vector search sometimes misses the mark for specific keywords. Is it possible to implement a hybrid search approach in Chroma DB that combines semantic vector similarity with traditional keyword-based BM25 filtering for better context retrieval?
3 answers
While Chroma DB is primarily a vector store, you can achieve hybrid search by utilizing its robust metadata filtering capabilities alongside vector queries. The strategy involves pre-calculating keyword tags and storing them as metadata strings. When a user queries, you perform a semantic search but apply a where filter that includes the specific keywords. I used this for a technical documentation bot; by filtering the search within specific 'tags' or 'categories' stored in the metadata, the accuracy of the retrieved chunks increased by 35%. For true BM25 integration, you may need to run a secondary search in a tool like Elasticsearch and then re-rank the results.
Donna, does adding complex metadata filters to every Chroma DB query significantly increase the latency compared to a raw vector similarity search?
Using a "cross-encoder" model to re-rank the top 10 results from Chroma is another way to simulate the benefits of hybrid search.
Sandra makes a great point. Re-ranking is a standard trick in Chroma DB workflows to ensure the most semantically relevant information is passed to the generative model.
Thomas, the latency hit is actually quite minimal! Chroma DB is optimized for metadata filtering during the vector traversal. In our benchmarks, adding a filter only added about 3-5ms to the total response time. The benefit of having a highly relevant answer for the LLM far outweighs this tiny delay in the retrieval phase.