Standard metrics like accuracy don't seem to work for creative text tasks. When fine-tuning generative AI models, what benchmarks should I be looking at to ensure the quality is actually improving? I’m looking for something more rigorous than just "vibes" and manual spot-checking.
3 answers
You should move away from BLEU or ROUGE scores, as they are often misleading for generative AI models. Instead, look into "LLM-as-a-judge" frameworks like G-Eval. This involves using a more powerful model (like GPT-4) to grade the outputs of your fine-tuned model based on specific criteria like coherence, relevance, and fluency. You can also measure "Perplexity" to see how well the model predicts the next token in your specific domain. Combining these automated metrics with a small, high-quality human evaluation set is currently the industry standard for a balanced view of model performance.
Are you focusing more on the "safety" and "toxicity" metrics for your generative AI models, or is the linguistic quality your primary concern right now?
A/B testing two different versions of your generative AI models with real users is still the most reliable way to measure true business impact and satisfaction.
Agreed. No matter what the automated score says, user retention and conversion are the final metrics that actually matter for the company.
My primary concern is the technical accuracy of the advice the model gives. I'll have to look into G-Eval to see if I can automate the fact-checking part of the grade.