We recently switched from OpenAI's text-embedding-3-small to a fine-tuned local model. Our Chroma DB retrieval metrics seem "off." Has anyone used the official Chroma Generative Benchmarking tool to detect semantic drift? How do you interpret the Recall@10 scores when your ground truth is constantly evolving?
3 answers
The Chroma benchmarking tool is excellent for this. It uses an LLM-as-a-judge to identify if the retrieved documents actually contain the information needed to answer the query. When you switch models, you should look at the Recall@k distribution. If your new model has a high cosine similarity but low recall, it means your vectors are "clumping" too much in the latent space. I usually run a test set of 1,000 queries through both models and compare the NDCG scores—if the new model is more than 5% lower, we stick to the old one.
Are you re-indexing your entire collection when you switch models? You can't compare embeddings from two different models in the same space!
Try visualizing your clusters with UMAP before and after the switch. It's the most intuitive way to see if your new embeddings are separating concepts correctly.
Visualizing the Chroma collection really helped us realize our new model was treating "Python" (the language) and "Python" (the snake) as the same entity. Back to the training board!
Yes, Jason, we did a full re-index. The issue is that the "semantic neighborhood" of our technical terms changed, which is why we’re looking at the Chroma tools to quantify the impact on our RAG pipeline.