The first time someone on a team I was advising said "we need to write tests for the LLM," I said yes without thinking carefully about what that meant. Everyone in the room nodded. Tests: good. We know how to write tests. We have coverage metrics. We have CI. This would be fine.
It was not fine. Six weeks later the team had three hundred "tests" that all passed, a model that was clearly getting worse on the cases that actually mattered, and no way to explain the discrepancy. The problem wasn't the runner or the tooling or the test coverage. The problem was that we were writing tests when we should have been writing evals, and the two things are fundamentally different in ways that matter from day one.
The fundamental difference
A unit test has a contract: given this input, assert this output. The output is known before the test is written. The test either passes or fails. There is no probability, no distribution, no tolerance. A passing test suite means the system behaves exactly as specified.
An eval asks a different question: given this input, is the model's response acceptable? "Acceptable" is not a value. It's a judgment. It's probabilistic — the same input run a hundred times will produce a distribution of outputs, some of which are acceptable and some of which are not. It's context-dependent — what's acceptable for a customer-facing support bot is different from what's acceptable for an internal code review assistant. And it degrades over time — the bar for "acceptable" tends to rise as the product matures and users develop higher expectations.
The practical implication: you cannot write evals before you've defined acceptability. And defining acceptability is a product decision, not a technical one. It requires someone to say, in concrete terms, what a good output looks like — not just for the routine cases, but for the adversarial cases, the edge cases, and the cases that the original spec didn't anticipate.
Why the three hundred tests all passed
When engineers trained on traditional testing write evals, they do what they know: they assert exact or near-exact matches. The model produces "The user's account was created successfully." The test asserts that the output contains "successfully." Pass. But what the test doesn't capture: did the model also suggest the user check their spam folder for a verification email? Did it include a link that will be wrong in three months? Did it respond in the user's language? Did it maintain the right tone for the context?
These are not nitpicks. They are the things users actually notice. String matching is cheap to write and nearly useless for evaluating language model outputs in any context where the output is meant to be read by a human.
What makes a good eval
The dataset is the hardest part of an eval suite, not the runner. The runner is fifteen lines of Python. The dataset is where your product judgment lives, and it requires you to think carefully about four categories of case:
- Routine cases. The 80% that should always work. Fast, cheap to add. Easy to over-index on.
- Adversarial cases. Inputs that look reasonable but expose failure modes. Requires adversarial thinking. Most teams skip this until they encounter the failure mode in production.
- Regression cases. Inputs that broke in the past. Every production incident should produce at least one regression case before the incident is closed.
- Boundary cases. The edges of what the system is supposed to handle. Where does the model's responsibility end? What happens when the user asks something the system isn't designed for?
Most teams build mostly routine cases and call it done. A suite that is 90% routine cases tells you almost nothing interesting. The distribution of case types in your eval suite should reflect the distribution of ways the system can fail, not the distribution of ways it's likely to be used correctly.
The judge paradox
Once you have a dataset, you need a way to judge outputs. The three options — human labelers, rubric-based scoring, and judge models — each have failure modes that are worth understanding before you commit to one.
Human labelers are the gold standard and the bottleneck. They're slow, expensive, and don't scale to continuous evaluation. They're essential for building the dataset and for calibrating the judge, but they can't be the primary judge for routine regression runs.
Rubric-based scoring is fast and cheap and breaks on anything outside the rubric. It works well when the output space is narrow and well-defined — classifying outputs into categories, checking for specific required elements — and fails badly when the output space is open-ended.
Judge models are powerful and introduce their own class of problem. The judge model has its own failure modes, which may overlap with the model under test if they share a training distribution. It can be inconsistent across runs. It can be gamed inadvertently — if you tune the model under test against the judge model's feedback, you're optimising for the judge's biases, not for actual quality.
Evals degrade and need maintenance
The thing nobody mentions when introducing eval frameworks: evals go stale. The dataset that was adversarial in month one becomes routine by month six. The model improves, or the prompts are tuned around the failure modes, and suddenly a suite that was genuinely challenging is just a green light that tells you nothing.
Evals need maintenance the same way integration tests do. They need someone responsible for keeping the adversarial cases genuinely adversarial, for adding regression cases when production incidents happen, and for reviewing whether the rubric still reflects what the product is actually trying to do. The teams that set up evals in month one and never touch them again are the teams that get surprised six months later by failures the eval suite should have caught.
A practical way to start
If you're starting from scratch: don't start with a framework. Start with a spreadsheet. Twenty-five to thirty cases, hand-written, covering each category. Judge them yourself against a rubric you wrote. Do it every time you change the model or the prompt. That discipline — sitting with the outputs and making a judgment, case by case — will teach you more about what your system actually does than any automated framework will in the first month.
Once you understand what you're measuring, automate the parts that are worth automating. But understand them first. The framework should serve the judgment, not replace it.