How to evaluate an LLM feature before it reaches users
A demo that works once is an anecdote. Here’s how we turn “it seemed fine” into numbers you can ship on.
- Write down what “good” looks like before you write the prompt.
- Build a golden set from real traffic, then grade it with code, a calibrated LLM judge and humans.
- Track latency, cost and refusals as hard as accuracy.
- Make the evals a gate in CI, so a model upgrade can’t quietly make things worse.
Every LLM feature has a magical first demo. The model summarises the contract, drafts the reply, answers the awkward question, and someone in the room says the two most expensive words in software: ship it.
Then real users arrive. They bring typos, half-sentences, three languages in one message and a screenshot of a PDF of a fax. The feature that dazzled on Tuesday is quietly embarrassing by Friday. The model usually isn’t the problem. The problem is that nobody wrote down what “good” means, so nobody can tell when it stops being good.
Here’s the evaluation playbook we use before an AI feature gets anywhere near a customer.
1. Write the job description before the prompt
Treat the feature like a new hire. What is the task? What inputs will it see? What does a great answer look like, what’s acceptable, and what must it never do? If you can’t grade an answer, you can’t improve it. You can only argue about it in Slack.
A one-page spec with three or four worked examples per grade saves weeks. It also makes the next step possible.
2. Build a golden set from real life, not your imagination
Start with 50 to 200 examples and label the outcome you expect for each. Pull them from support tickets, search logs, sales calls and anywhere else real people have already asked for what the feature will do. Cover four kinds:
- The boring majority: the everyday requests that make up most of the traffic.
- Edge cases: long inputs, empty inputs, mixed languages, ambiguous asks.
- Adversarial cases: people trying to make it misbehave.
- Out of scope: things it should politely decline.
Then refresh the set every month. Production is a moving target, and a golden set that never changes slowly becomes a museum.
3. Grade at three levels
No single grader is good enough, so we stack them, from cheapest to most expensive:
- Deterministic checks. Valid JSON, required fields present, length limits, citations included, no personal data in the output. Fast, cheap, and they never get tired.
- A model-graded rubric. An LLM judge scores each answer against your written rubric. Before you trust it, have humans grade a sample too and check how often they agree.
- Human review for the slice that matters most: high-value customers, regulated topics, anything with money attached.
An LLM judge is a junior reviewer: fast, useful, and in need of supervision.
4. Measure what users feel, not just what’s correct
Accuracy is the headline, but users experience the whole package. Track these alongside it:
- Latency at the 95th percentile, not the average. A correct answer in 14 seconds is a wrong answer in a checkout flow.
- Cost per task, so a clever prompt chain doesn’t become a surprise invoice.
- Refusal rate: too high and the feature is useless, too low and it’s reckless.
- Grounding: when the answer should come from your documents, how often it invents something instead.
- Escalation rate: how often a human has to step in.
5. Red-team it like it owes you money
Try prompt injection hidden in retrieved documents, jailbreak attempts, off-topic requests and attempts to pull out other users’ data. Every failure you find goes into the golden set as a permanent test. Guardrails are part of the feature, not a ticket for next quarter.
6. Make evals a gate, not a ceremony
Every prompt change, retrieval tweak and model upgrade runs the suite, and a regression blocks the merge. Model upgrades deserve special suspicion: the new version is usually better on average and occasionally worse on exactly the thing your users care about.
If a change can’t show you a before-and-after score, it isn’t ready for review.
7. Ship behind a dial
Go live in stages. Run it in shadow mode, where the model answers but nobody sees it. Then release it to internal users, then to a small percentage of customers. Log inputs and outputs, with consent and with personal data removed, and send every failure back into the golden set. The feature gets better because the test gets better.
The goal isn’t a perfect model. It’s knowing, before your users do, exactly how imperfect it is, and deciding that’s good enough on purpose.
