My RAG system keeps making up facts even when the context is provided. I'm looking into the Guardrails AI 'provenance' and 'hallucination' validators. Does anyone have experience setting these up? I specifically want to know how to link the validator to my vector database results so it can verify the grounding of the answer.
3 answers
To stop hallucinations, you need to use a validator that performs "Self-Correction" or "Re-asking." In your RAIL specification, you can define a validator that compares the LLM's response against the retrieved chunks from your vector store. Guardrails AI can use a Natural Language Inference (NLI) model to check if the response is "entailed" by the context. If the score is low, the guard can automatically send the prompt back to the LLM with instructions to stick closer to the provided documents. This iterative loop is very effective, though it does increase the token cost, so you should monitor your API usage closely when this is enabled.
Are you passing the retrieved context as metadata to the guard object? The validator needs that specific key-value pair to perform the grounding check.
You should also check out the 'Correct' strategy. It can strip out the unverified parts of the sentence while leaving the truthful parts intact for the user.
That’s a great suggestion, Natalie. It’s much better than a total failure message when only one small part of the response is off-base.
Ah, that might be what I’m missing. I was passing the context in the prompt but not in the metadata field of the Guard call. I didn't realize the validator looked for a separate metadata object to do the comparison. I'll restructure my Python code to pass the 'sources' explicitly to the guard today.