Evals Are the Only Thing Standing Between You and a Regression

Here is a scene we have walked into more than once. A team has an LLM feature in production. Someone tweaks the system prompt to fix a complaint from a single customer. It ships. Two weeks later, a different behavior has quietly broken, nobody knows when, and the only way to find out what changed is to read the git log and guess.

That team does not have a prompt problem. They have a measurement problem. Every change they make is a guess with a deploy attached, and the feedback loop runs through customer complaints.

Evals are how you close that loop. They are not a nice-to-have you add once the feature is mature - they are the thing that makes the feature changeable at all.

Without evals, "we improved the prompt" is a vibe. With evals, it is a number you can defend.

---

What an Eval Actually Is

An eval is three things: a set of inputs, a definition of what a good output looks like, and a scoring function that turns the gap between them into a number. That is it. The rest is engineering discipline.

The reason evals feel hard is that the middle part - defining "good" - is genuinely hard for open-ended text. Teams stall there, decide the problem is unsolvable, and go back to eyeballing outputs in a playground. The way out is to stop trying to score the whole output at once and start scoring the specific properties you actually care about.

For a support agent, you probably care that it retrieved the right policy document, that it did not promise a refund it has no authority to give, that it escalated when it was unsure, and that the tone was right. Those are four separate checks with four different scoring methods. Bundling them into one "is this response good?" judgment throws away the information you need to debug.

---

Build the Eval Set Before You Build the Feature

The single highest-leverage habit we have is writing the eval set first. Not the full suite - twenty to fifty cases is enough to start. Doing it first forces you to be specific about what the feature is supposed to do, which surfaces disagreements about scope while they are still cheap to resolve.

Where the cases come from, in rough order of value:

• Real production traffic. Once you have any, this dominates everything else. Sample broadly, then oversample the weird tail: long inputs, unusual formats, users who type in two languages.

• Recorded failures. Every bug report becomes a permanent eval case. This is the single best habit a team can build. A bug you fixed without adding a case is a bug you will ship again.

• Adversarial cases. Prompt injection attempts, out-of-scope requests, requests for things the system must refuse. These do not come from real traffic often enough to catch by sampling.

• Synthetic cases. Useful for coverage of combinations you know are possible but have not seen. Weakest signal of the four, because you are testing against your own imagination.

Keep the set version-controlled next to the code. It is a test fixture, not a spreadsheet in a shared drive.

---

Scoring: Pick the Cheapest Method That Works

There is a hierarchy here, and teams routinely reach for the most expensive option first.

Deterministic checks. Did it return valid JSON? Does the response contain a citation? Did it call the right tool with the right arguments? Is it under the length limit? These are free, instant, and never flaky. A surprising fraction of what you care about is checkable this way, and it is always worth asking "can I assert this in code?" before reaching for anything smarter.

Reference-based scoring. When there is a correct answer - an extraction task, a classification, a retrieval step - compare against ground truth. Exact match, F1 over extracted fields, recall at k for retrieval. Cheap, reliable, and directly interpretable.

Model-graded scoring. For genuinely open-ended output, use a model as a judge. This works, but only with discipline: give the judge a rubric with concrete criteria rather than "rate this 1-10", have it output structured scores per criterion, and validate the judge against human labels before you trust it. An unvalidated judge is just a second model with an opinion.

Human review. The most expensive and the most trustworthy. Reserve it for calibrating the automated scorers and for periodic spot-checks, not for every run.

The mistake we see most often is jumping straight to a model judge for things a two-line assertion would have caught more reliably and for free.

---

Run Them in CI or They Will Rot

An eval suite that runs when someone remembers to run it is not a safety net. It is a document about intentions.

We wire evals into CI the same way we wire unit tests. A pull request that touches a prompt, a retrieval config, a tool schema, or a model version triggers the suite. The results post to the PR as a diff against the base branch: which cases regressed, which improved, and what the aggregate scores did.

A few practical constraints make this sustainable:

Keep a fast tier and a slow tier. A fifty-case smoke suite runs on every commit in under two minutes. The full suite, which might be several hundred cases with model-graded scoring, runs on merge to main or nightly. If the fast tier takes ten minutes, people will start skipping it.

Cache aggressively. Eval runs are expensive in tokens. Cache by hash of prompt plus input plus model version, and you only pay for what actually changed.

Fail on regression, not on absolute score. An eval suite that requires 95 percent to pass will get its threshold lowered the first time it blocks a release. A suite that fails when a specific case that used to pass now fails gives you a real signal that someone has to look at.

---

The Metrics That Matter

Accuracy is the obvious one and rarely the most important. The metrics we track on almost every agent we ship:

• Task success rate - did it accomplish the thing, scored per task type rather than in aggregate. Aggregate numbers hide the fact that one category fell off a cliff.

• Escalation rate - how often it handed off to a human. Both directions are bad. Too high and the automation is not earning its keep; too low and it is guessing when it should be asking.

• Refusal accuracy - did it refuse the things it should refuse, and only those. This one drifts more than teams expect when prompts change.

• Latency at p95 - the average is a lie when tool calls and retries are involved.

• Cost per task - tokens in, tokens out, per successful completion. A change that improves accuracy by two points and triples the cost is a decision, not an improvement.

Track all of these per release, not just at the moment you are debugging something. The trend line tells you things a single run cannot.

---

Evals Change What You Are Willing to Try

The underrated benefit is not catching regressions. It is that a good eval suite makes you brave.

When a change costs nothing to validate, you try more of them. You swap the model to a cheaper one and find out in ten minutes that it holds up on 90 percent of your cases, so you route the easy ones to it. You restructure a bloated system prompt because you can prove nothing broke. You accept a contribution from someone who does not have the whole system in their head, because the suite has an opinion about whether it works.

Teams without evals stop touching the prompt. It becomes a load-bearing artifact nobody understands and everyone is afraid of. We have inherited several of these, and unpicking them always starts the same way: build the eval set, establish a baseline, then start changing things.

---

Failure Modes We See in Eval Suites

A suite can exist and still not be doing its job. The recurring problems:

The set is too easy. If everything passes on every run, the suite is not discriminating between good and bad versions. A healthy suite has a handful of cases that sit near the boundary and flip when something meaningful changes. When we inherit a suite passing at 100 percent, the first thing we do is go find the hard cases it is missing.

Cases duplicate each other. Forty variations of the same customer question measure one thing forty times. Coverage means distinct failure modes, not volume. We would rather have sixty genuinely different cases than four hundred near-duplicates, and the smaller set runs fast enough that people use it.

The ground truth is wrong. Expected outputs written quickly, months ago, by someone who has since left. When a case fails, the first question has to be whether the model is wrong or the label is. Suites where nobody trusts the labels get ignored entirely, which is worse than not having them.

The judge drifts. If your model-graded scorer is itself an LLM, changing its model version changes your scores with no change to the system under test. Pin the judge version explicitly and treat upgrading it as its own change, with its own validation against human labels.

Nobody looks at the outputs. Aggregate scores hide a lot. We make a habit of reading the actual outputs of a random sample every time the suite runs on main, because there are failure modes - a tonal shift, a subtle formatting change, a new habit of hedging - that no scorer was written to catch yet.

---

Evals for Agents Are Different

Everything above assumes a single input produces a single output. Agents that take multiple steps and call tools need more.

Score the trajectory, not just the result. An agent that reaches the correct answer after eleven tool calls, three of them wrong, is not working - it got lucky. We assert on the path: did it call the expected tools, in a sensible order, without redundant calls. A trajectory that degrades while the final answer stays correct is an early warning of a regression that will surface as a failure later.

Mock the tools deterministically. An eval that hits real APIs is slow, flaky, and occasionally destructive. Record real tool responses once and replay them, so the only non-deterministic component in the run is the model. This is the difference between a suite you can run on every commit and one you run monthly.

Test the failure paths explicitly. What does the agent do when a tool returns an error, times out, or returns an empty result? These are the cases that break in production and the ones almost never present in an eval set built from happy-path traffic. We write them by hand because they do not show up by sampling.

Include cases that should stop. An agent that should recognize it lacks the information to proceed and ask a question, but instead invents an answer, is failing in the most expensive way available. Cases where the correct behavior is to escalate deserve as much coverage as cases with a correct answer.

---

Where to Start

If you have an LLM feature in production and no evals, do this in order. Pull fifty real inputs from your logs. Write down what a good output looks like for each - actual expected values, not adjectives. Add deterministic assertions for everything you can check in code. Add a model judge, with a rubric, only for what is left. Wire it into CI. Then add every future bug report as a case, permanently.

That is a couple of days of work and it changes the economics of every subsequent change you make.

If you are building agents that take real actions against production systems, the eval suite is not separate from the architecture - it is part of it. Our AI Agent Development practice treats it that way, alongside the tool design and guardrails covered in Architecting Production LLM Agents.

You do not need a perfect eval suite. You need one that fails when you break something, and you need it before you break something.