Even with Retrieval-Augmented Generation, I'm seeing cases where the AI ignores the provided context and makes things up. What are the best strategies to force the model to stay grounded? Should I focus on the prompt engineering side or the vector database retrieval quality?
3 answers
You need to tackle this from both ends. First, improve your retrieval by using "Hybrid Search" (combining semantic vector search with keyword-based BM25). This ensures the most relevant documents are actually in the context. Second, use "Chain of Verification" (CoVe) in your prompts. Tell the model to first extract facts from the context, then verify them, and only then write the final answer. You can also add a system prompt instruction like: "Answer ONLY using the provided context. If the answer isn't there, say you do not know."
Have you tried adjusting the "temperature" setting of your LLM to a lower value like 0.1 to make the output more deterministic?
Implementing a "Citation" requirement helps; ask the model to provide the specific source line for every claim it makes.
I agree, Susan. This "attribution" method makes it much easier for the end-user to spot when the AI is hallucinating versus quoting the source.
Lowering the temperature is a great quick fix, Richard! It reduces the randomness of the token selection. However, if the retrieved data is garbage, a low temperature will just produce "confidently incorrect" garbage. Quality of data is still king.