How to evaluate an AI feature before you ship it (evals for non-researchers)
You don't need a research team to test an LLM feature properly. A practical, data-backed guide to building a small evaluation suite that tells you if your AI actually works — with real-world examples of what happens when teams skip it.
Here’s how most teams test an AI feature: someone types five questions into it, the answers look good, it ships. Then real users type the five thousand questions nobody tried, and quality becomes a matter of anecdote and vibes.
That gap between “looked good in the demo” and “held up in production” is not a rounding error. Research firm RAND puts the enterprise AI project failure rate above 80% — roughly double the failure rate of ordinary IT projects — and MIT’s Project NANDA found that around 95% of generative AI pilots never produce a measurable financial return. A 2026 survey by Applause of software, QA, and AI professionals found that for most organizations, fewer than half of their AI initiatives make it from proof-of-concept to production at all. These aren’t stories about bad models. They’re stories about teams that never built a way to measure “good” before they shipped.
The fix is an evaluation suite — an “eval” — and despite the research-paper aura around the word, building a useful one is well within reach of any product team. Here’s the practical version, with the data and the real incidents that make the case.
What actually happens when there’s no eval
In February 2024, a Canadian tribunal ruled that Air Canada was liable for advice its support chatbot gave a customer named Jake Moffatt. Moffatt asked about bereavement fares after his grandmother died, and the bot told him he could book a full-price ticket and claim a retroactive discount within 90 days. No such policy existed — the bot had confused two different pages on Air Canada’s own site. Air Canada argued the chatbot was responsible for its own words; the tribunal called that “a remarkable submission” and ordered the airline to pay damages. The company’s defense, in effect, was “we never actually checked what our bot says.”
That failure mode repeats. In April 2025, a support bot at the coding tool Cursor invented a device-limit policy that didn’t exist and told a confused customer about it as fact — the company’s co-founder had to publicly confirm the policy was fabricated. Neither case involved a model that was incapable of getting things right. Both involved a system that was never systematically checked against realistic questions before real customers relied on its answers.
This is the pattern an eval exists to catch — not “is the model smart,” but “does this specific feature, on the questions our real users actually ask, say things we’re willing to stand behind.”
The numbers behind the failure rate
A few data points worth sitting with before you decide an eval is optional:
- 80.3% of enterprise AI projects fail to deliver their intended business value, according to RAND’s 2025 analysis of more than 2,400 enterprise AI initiatives — roughly double the failure rate of comparable non-AI IT projects.
- 42% of companies abandoned most of their AI initiatives in 2025, up sharply from 17% the year before, per S&P Global Market Intelligence’s Voice of the Enterprise survey.
- 88% of AI agents that get built never reach production, but the ones that do reach production and stay there return an average 171% ROI — the gap between the two groups is overwhelmingly a matter of evaluation, monitoring, and governance discipline, not raw model capability.
- In Applause’s 2026 survey of AI and QA professionals, teams repeatedly said they were “conducting evals” but lacked the methodology and inter-rater consistency to make those evaluations mean anything at scale — producing scores that felt credible without reflecting how the system actually behaves with real users.
None of this says the technology doesn’t work. It says most teams underestimate the engineering rigor it takes to get from an impressive demo to something reliable enough to put in front of customers — and evaluation is the single highest-leverage piece of that rigor, because it’s the thing that tells you, before launch, whether you’re in the confident minority or the unmeasured majority.
What an eval actually is
A list of real inputs, what a good output looks like for each, and a way to score the AI’s attempt. That’s it. Think of it as unit tests for a system that’s allowed to be a little different every time.
With one in place, questions that used to be arguments become measurements. Should we switch to the cheaper model? Run the eval. Did last week’s prompt change break anything? Run the eval. Is the new version actually better or does it just feel better? Run the eval.
Building one in four steps
1. Collect 100–300 real examples. Not invented ones — real tickets, real queries, real documents from the actual workflow. Include the awkward cases: vague questions, typos, angry customers, edge cases from your worst data.
Take the Air Canada case as the template for what belongs in your set. “What’s your bereavement fare policy?” is the obvious question. The one that actually broke things was the follow-up: “Can I apply for the discount after I’ve already booked and traveled?” — a question that touches two different policy pages at once. If your eval only contains polite, single-topic, well-formed inputs, it will happily approve a feature that fails exactly the way Air Canada’s did: on the question that straddles two sources of truth.
2. Define “good” for each. For classification-type tasks (routing, tagging, extraction), that’s just the correct answer. For generation tasks (drafts, summaries, policy answers), write down the criteria instead: must mention X, must not promise Y, must say “I’m not sure, let me check” rather than invent an answer when the policy isn’t in its source material, matches our tone, under 150 words. Concrete criteria beat “sounds good” — and “admits uncertainty instead of fabricating” deserves to be its own explicit pass/fail criterion, not an assumption.
3. Score it — with code where possible, with a model where not. Exact answers can be checked automatically. For prose, the standard trick is using an LLM as the judge: give it the output and your criteria, ask for a pass/fail with a reason. It’s imperfect but consistent — and you calibrate it once by checking a sample of its judgments against a human’s. Applause’s own approach at enterprise scale is a useful upper bound to know exists: they run at least three independent frontier models from different providers on each output in parallel, specifically to avoid one vendor’s blind spots dominating the score, and still route disagreements to a human reviewer. You don’t need that much machinery to start — a single LLM judge checked against human spot-review gets a small team most of the value.
4. Set a bar and automate the run. Decide what’s shippable — say, 95% on routing accuracy, 90% on draft quality, 100% on “never invents a policy that isn’t in the source documents” — and wire the eval to run on every prompt or model change. A one-command script is enough. The point is that no change reaches users without a score.
A worked example
Say you’re building a support bot that answers billing questions from a knowledge base, and refund policy is one of the trickiest areas — the exact shape of the Air Canada problem. A minimal eval for just that slice might look like:
| Input | What “good” looks like | How it’s scored |
|---|---|---|
| “Can I get a refund if I cancel within 24 hours?” | States the actual 24-hour policy, cites the source article | Exact-match against known policy text |
| “My flight was delayed 6 hours, do I get a refund AND a rebooking credit?” | Doesn’t combine two separate policies into one that doesn’t exist | LLM judge checks for policy conflation |
| “I already used the discount, can I get it again retroactively?” | Says no, or says “I don’t have information on that, please contact support” | LLM judge checks for fabricated permission |
| (a typo-ridden, angry message with no clear question) | Asks a clarifying question rather than guessing | LLM judge checks it didn’t answer a question that wasn’t asked |
That fourth row matters as much as the first. A bot that’s 98% accurate on clean questions and confidently wrong on the ambiguous 2% is exactly the profile that ends up in a tribunal ruling — the failures aren’t spread evenly, they cluster in the edge cases nobody thought to test.
The habits that keep it useful
- Feed failures back in. Every bad output someone catches in production becomes a new eval case. Your suite grows into a map of exactly how your feature likes to fail — which is precisely what you want to test. If Air Canada had done this after the first customer complaint about bereavement fares, there wouldn’t have been a second one that ended up in front of a tribunal.
- Keep a held-back set. If you tune your prompt against the same examples over and over, you’ll overfit to them. Keep a slice you rarely look at for honest final checks.
- Track the score over time. Model providers update models; your data drifts. A monthly eval run catches quiet regressions before your customers do.
What this costs, honestly
The first eval takes a few days of focused work, most of it collecting examples and arguing about what “good” means. That argument is the hidden benefit — half the teams we work with discover mid-eval that they never actually agreed on what the feature was supposed to do.
A few days feels expensive next to “just ship it.” It’s cheap next to the alternative: a support bot that invents a policy, a customer who takes you at your word, and a bill — reputational or literal — that arrives months later with your company’s name on it.
Sources: RAND Corporation, “Why AI Projects Fail” (2025); MIT Project NANDA, “State of AI in Business” (2025); S&P Global Market Intelligence, Voice of the Enterprise survey (2025); Applause, “State of Digital Quality in AI” (2026); Moffatt v. Air Canada, 2024 BCCRT 149; PYMNTS reporting on Cursor’s April 2025 support bot incident.