Accuracy is our top priority for a medical search tool we are building. What are the best practices for evaluating Haystack for production RAG systems to ensure we are minimizing hallucinations? We need to track Faithfulness and Context Precision metrics over time. Does the framework provide built-in tools for batch evaluation, or should we be looking at integrating external libraries like RAGAS?
3 answers
For a high-stakes domain like medical search, you definitely need a rigorous evaluation framework. Haystack has built-in Evaluation nodes that allow you to compare your pipeline's output against a "Golden Dataset." I recommend setting up an automated pipeline that runs every time you update your prompt or model. You can calculate metrics like Mean Reciprocal Rank (MRR) for your retriever and use an LLM-based evaluator to check for groundedness. Integrating RAGAS is also quite straightforward since Haystack’s output format is very clean, making it easy to pass data into specialized evaluation libraries.
How are you handling the generation of your ground-truth datasets for these evaluations? It's often the hardest part of the process. Are you using a synthetic data generation approach with a stronger model like GPT-4, or are you relying on human experts to manually label the correct document-answer pairs for your medical use case?
Focus on the "Retriever" first. If the context isn't right, the answer won't be either. Use Haystack's Recall and Precision metrics to tune your top_k parameters before fixing the generator.
That's a great tip, Susan. We found that increasing top_k from 5 to 10 significantly improved our faithfulness score because the generator had more relevant medical context to work with.
For our medical project, we used a hybrid approach. We used GPT-4 to generate initial question-answer pairs from our corpus, but then had medical students verify them. It's too risky to rely purely on synthetic data for evaluation in healthcare. We then fed this verified set into the Haystack EvalRun class to get our baseline metrics before we ever touched the production environment.