I've implemented a Retrieval-Augmented Generation pipeline using LangChain and Milvus. Even though I'm retrieving the "relevant" documents, my LLM still gives wrong answers or ignores the context provided. I'm using cosine similarity for the search. Could the issue be with my chunking strategy, the embedding model, or the way I'm ranking the results before feeding them to the prompt?
3 answers
Check your chunking! If you cut off sentences in the middle, the embedding loses its meaning. Use a recursive character splitter to keep paragraphs intact.
Hallucinations in RAG often stem from "semantic overlap" where the top-k retrieved chunks are similar in language but don't contain the actual answer. First, evaluate your chunk size; if chunks are too small, they lose context. Second, look into a "Reranker" (like Cohere or BGE). Vector search is great at broad retrieval, but a reranker is better at fine-grained relevance. By taking the top 20 results from Milvus and passing them through a reranker to pick the best 5, you ensure the LLM gets the most precise information, drastically reducing the chance of it making up facts.
Are you using a hybrid search approach? Sometimes keyword-based BM25 search finds specific technical terms that vector search might smooth over.
Steven is onto something. Pure vector search can be "too fuzzy." If a user asks for a specific part number or a niche error code, a vector database might return a general tutorial instead. This is why hybrid search—combining dense vectors with sparse keyword vectors—is becoming the standard. It gives you the semantic understanding of embeddings plus the literal precision of traditional search. Most modern vector DBs now support this natively, and it's often the single biggest fix for RAG accuracy issues.
Great point, Betty. Adding a bit of "overlap" between chunks also helps ensure that the context isn't lost at the boundaries of the split.