We’ve launched our AI search engine using Haystack, but management wants to see concrete metrics on its performance. What is the best way to use the "Evaluation" module in Haystack to track Mean Reciprocal Rank (MRR) and Recall? Can we automate this evaluation to run every time we update our document corpus or change a model?
3 answers
Haystack has a very robust eval mode specifically for this. You can create a "Label" dataset containing pairs of questions and the correct document IDs. By running your pipeline through the eval() method, the framework automatically calculates metrics like Recall@k and MAP (Mean Average Precision). In early 2024, my team integrated this into our CI/CD pipeline. Every time a developer pushed a change to the retriever settings, Haystack would run the evaluation against our "Golden Set" and block the deployment if the MRR dropped below our 0.80 threshold. It’s essential for maintaining quality.
Do those automated evaluations also cover the "Generator" part of the RAG pipeline, or just the retrieval? It’s one thing to find the right document, but it's another thing entirely for the AI to summarize it correctly without hallucinating facts.
The Pipeline.eval() function is straightforward and outputs a nice report that you can even export to a pandas DataFrame for further analysis and visualization for stakeholders.
Exactly! Being able to present a clear graph of Recall improvements over time is the best way to prove the ROI of your AI search engine project to management.
Keith, for the generator part, Haystack now supports "Model-based Evaluation." You can use an LLM (like GPT-4) as a judge to score the final answer based on criteria like "Faithfulness" and "Relevancy." It’s basically the RAGAS framework integrated within Haystack. This gives you a complete picture of the search engine's health—from the initial query to the final answer—allowing you to pinpoint exactly where the system needs improvement.