I’m a Business Analyst leading a new AI implementation for our customer service department. My stakeholders expect a "perfect human-like chatbot" by Q3, but they don't understand the concepts of "model drift," "training data requirements," or "probabilistic vs deterministic" outputs. How can I manage these expectations without getting bogged down in math? What are the best KPIs to show progress when the "accuracy" of the model is still being refined?
3 answers
Managing AI expectations is 80% communication. In my 2023 projects, I stopped talking about "F1 Scores" or "Loss Curves" and started talking about "Business Confidence Levels." Instead of saying the model is 85% accurate, I’d show them a "Confusion Matrix" translated into business terms: "Out of 100 customers, the AI will handle 85 perfectly, and 15 will be gracefully handed to a human." For KPIs, focus on "Time to Resolution" or "Reduction in Human Tier-1 Tickets." This grounds the project in ROI rather than the "magic" of the tech.
How do you explain the concept of "hallucinations" to executives who think a computer shouldn't make up facts?
Start with a pilot! Showing a working prototype with 70% accuracy is much better than a PowerPoint promising 99% in six months.
Exactly! Once they see the "magic" in person, even with errors, they tend to become much more supportive and understanding of the development process.
William, I use the "Confident Intern" analogy. I tell them the AI is like a brilliant intern who knows everything on the internet but occasionally gets overconfident and makes things up. To prevent this, we explain that we are building "guardrails" (like RAG or strict prompting). This makes the technical limitation feel like a manageable risk rather than a fatal flaw. It helps to show them the "source citations" the AI uses to build their trust in the factual nature of the responses.