Ask any leadership team how their AI assistant is performing, and you’ll probably get an answer. Usage is up. Fewer customer touchpoints need escalation. Users seem to like it.
Now ask how they know it gives the right answers. Not most of the time. Every time it matters. That’s where the room usually gets quiet.
It’s no secret that AI is dominating the landscape today. Companies are shipping chatbots, voice assistants, and AI agents faster than they ever shipped websites or apps previously. Unfortunately, most are tested the way we tested websites in 2010: someone clicks around, it looks fine, and it goes live. That worked when software did the same thing every time. But AI doesn’t.
QA Playbooks Assume Software is Predictable
Traditional testing is built on a simple premise: the same input produces the same output. Look at the expected result, compare it to the acceptance criteria, and pass or fail.
AI breaks that contract in three ways:
1. It improvises. Ask the same question twice, and you may get two different answers. Both might be right. One might be confidently wrong.
2. It remembers … and forgets. A response can be perfect on its own and wrong in context, because turn six forgot what the customer said in turn two.
3. It can change without a release. A model update, a prompt tweak, or a new knowledge source can shift behavior overnight. There is no deployment or commit to trace back to, so no regression is run.
Simply put, traditional testing cannot judge any of that. If your test case’s acceptance criteria expects the bot to reply with a specific sentence, you’re not testing AI. You’re testing a script it will never follow.
Small Misses Become Big Failures
We’ve seen many teams test AI by channel alone. The voice team tests speech recognition. The conversation team tests intents. The app team tests the UI. Each one hits its target.
On the surface, that makes sense. It’s part of the classic split between unit, integration, and end-to-end testing. In traditional software, it works because every component has a contract with those defined inputs and defined outputs. If you honor the contracts and test the handoffs, then end-to-end testing is mostly just confirmation. That’s why the test pyramid puts thousands of unit tests at the bottom and a thin layer of end-to-end tests on top.
But AI layers don’t have contracts. They have probabilities. And probabilities multiply.
For example, a customer says “cancel my order” on a voice line. The speech engine gets it right 97% of the time. This time it hears “cancel my border.” It doesn’t throw an error. It hands the AI a perfectly valid sentence, so the handoff passes. It should recognize it can’t act on that and send the customer back down another path. But it was only ever tested on clean text, so it gives a polite non-answer instead. The app still shows the order as active. The customer calls back angry.
In this case, no layer failed its own tests. Each had a small miss rate the team signed off on. The problem is when you stack them, the misses compound:
• Three layers at 97% each produce a journey that works about 91% of the time.
• Add a fourth layer, and you’re near 89%.
• Each miss also hands the next layer an input that its tests never covered, so the real number is usually worse.
Nine out of ten sounds fine until you remember the tenth is a real customer.
Layer tests measure each miss rate in isolation. Only end-to-end testing measures what the customer experiences: all of them multiplied together.
In traditional QA, end-to-end testing confirms what the lower levels already proved. In AI, it’s the only place compounding shows up. That breaks the pyramid. End-to-end testing can’t be a thin layer on top. It needs as much investment as everything beneath it.
The pyramid was built on simple economics. End-to-end tests were slow to write, brittle to maintain, and expensive to run. Therefore, teams kept them to a minimum. AI and automation change that math. Journeys can be generated from requirements, heal themselves when screens change, and get scored by meaning instead of exact wording. Running them at the volume AI demands still takes time and compute. But the cost that limited end-to-end testing has collapsed.
Test the Journey, Judge the Meaning
Two shifts fix most of this.
First, test journeys, not just components. Run real user flows end-to-end and validate every layer at the same time: what was heard, what the AI decided, what the screen showed. When something breaks, you see exactly where. Fault isolation turns a vague “the bot was wrong” into a fixable defect with an owner.
Second, judge responses by their meaning, not their wording. The question isn’t whether the AI said the exact sentence you expected. It’s whether the answer was accurate, complete, relevant, and aligned with the business intent. That takes a rubric, a judge that can evaluate meaning (often including another model, checked by humans), and repeated runs. One pass proves nothing when the output varies. A pass rate across many runs tells you something real.
Know What You Tested and Admit What You Didn’t
Ask where your AI test cases come from. If the answer is “the team brainstormed some,” you have no idea what you’ve covered.
Anchor the testing to the requirements. Every requirement should get multiple scenarios, including different phrasings and different paths. Simulate the edge cases real people produce. Then map it. The requirements with zero scenarios are your actual risk register.
And be honest about the ceiling. A passing test shows the AI behaved well in the situations you tested. That’s it. “Never be rude” or “never give financial advice” can be tested hard, including with adversarial prompts designed to break it. But you can never fully prove it. Anyone who tells you their AI is guaranteed safe is selling something.
That candor isn’t a weakness. It’s how you decide where humans stay in the loop.
Three Questions for Monday
If you own an AI experience, ask your team:
1. When our AI gives a wrong answer, can we tell which layer caused it?
2. Which requirements have no test scenarios today?
3. What happens to quality the next time the model updates, and who notices first?
If the answer to the last one is “a customer,” you know who’s testing your chatbot.
We’ve put our full framework for end-to-end AI testing, across voice, chat, agents, and embedded AI, into a white paper. Want an early copy? Let’s talk.