I'm building a Retrieval-Augmented Generation (RAG) system for our technical manuals. My current setup often retrieves irrelevant chunks, causing the LLM to give wrong answers. Should I focus on changing my chunking strategy, or is the problem likely in the embedding model or the vector database retrieval settings? I'm using FAISS right now.
3 answers
Improving RAG usually requires a multi-pronged approach. First, try "Parent Document Retrieval"—instead of passing small chunks to the LLM, you retrieve small chunks but pass the larger parent document for better context. Second, look at "Self-Querying" retrievers if your manuals have metadata like version numbers or categories. FAISS is great for speed, but ensure you are using a high-quality embedding model like text-embedding-3-small. Finally, adding a "Reranking" step with a Cross-Encoder after initial retrieval can dramatically filter out those irrelevant chunks you mentioned.
Have you experimented with "Recursive Character Text Splitting" to ensure that your chunks don't break in the middle of a vital technical sentence?
Don't underestimate the power of a good system prompt. Sometimes telling the LLM to say "I don't know" if the context is weak is the best fix.
Spot on, Gregory. Guardrails in the prompt are just as important as the quality of the data coming out of the vector store.
Raymond, that change actually made a massive difference! Switching from a fixed-size splitter to the recursive one allowed the chunks to respect paragraph boundaries. The LLM now gets much more coherent snippets of information, which has almost entirely eliminated the hallucinations we were seeing earlier. It’s a simple change but incredibly effective for technical documentation where structure matters.