I see many teams deploying to production without any metrics beyond "it looks okay." Isn't the reality that most RAG systems are badly designed because they lack a feedback loop? How do you guys actually measure the "faithfulness" and "relevance" of your retrieved chunks?
3 answers
The "vibes-based" evaluation is exactly why most RAG systems are badly designed. Without a framework like Ragas or TruLens, you are flying blind. You need to measure the "RAG Triad": Context Relevance, Groundedness, and Answer Relevance. By using an LLM-as-a-judge, you can synthetically score thousands of queries to find where your retrieval is failing. A professional design includes a continuous monitoring pipeline that flags whenever the LLM starts answering based on its internal weights rather than the provided context, which is the most common failure mode.
Pamela, when using an LLM-as-a-judge, how do you deal with the "bias" of the evaluator model? Does a more powerful model like GPT-4o always give a more accurate score than a specialized smaller model?
If you aren't versioning your prompts and embedding models together, your system is likely to break silently over time.
Exactly, Melissa. We learned the hard way that changing an embedding model without re-indexing the entire database is a recipe for disaster. POSTED BY: Brandon Taylor
Joshua, cost and bias are real issues. We often use a "Panel of Judges" approach where we average scores from different models. Also, building a "Golden Dataset" of human-verified answers is the only way to truly calibrate your automated judges. It’s extra work, but it’s the only way to ensure your system isn't just a high-tech hallucination machine.