We are moving into the testing phase of our AI agent. I see that LlamaIndex has an evaluation module with classes like FaithfulnessEvaluator and RelevancyEvaluator. How do these actually work in practice? Can we automate this to run against a "golden dataset" of questions and answers to ensure our updates don't break the system?
3 answers
Evaluation is critical for avoiding "hallucinations" in production. The FaithfulnessEvaluator checks if the answer is actually derived from the retrieved source text, while the RelevancyEvaluator checks if the answer actually addresses the user's query. In late 2023, I implemented a pipeline where we used a separate "Judge LLM" (like GPT-4) to score our smaller RAG model's outputs. LlamaIndex allows you to wrap this into a BatchEvalRunner, which can process hundreds of test cases automatically. This is the only way to get objective "scores" on whether your chunk size changes are helping or hurting.
Are you looking at "Context Recall" as well, or just the final answer quality? Sometimes the LLM gives a perfect answer, but it’s only because it already knew the info from training, not because the RAG system actually found the right document.
You should also check out the integration with RAGAS through LlamaIndex. It provides even more granular metrics like "Answer Correctness" and "Context Precision."
Exactly, Ashley! RAGAS plus LlamaIndex is the gold standard for testing. It provides a comprehensive dashboard view of your system's health before you go live.
Paul, that's exactly why LlamaIndex’s RetrieverEvaluator is so important. It measures if the ground-truth document was even in the top-k results. If you don't measure the retrieval separately from the generation, you'll never know which part of your pipeline needs fixing. Automating this with a small script that runs every time you change your embedding model is a best practice for any AI team.