A few years ago I was studying algorithmic trading and working through a trend-following strategy. One of the first things you learn in that world is: you never go live before you've backtested. You don't trade with real money until you've run the strategy against historical data it's never seen, measured whether your edge holds up, and understood exactly what failure looks like — not in theory but in specific numbers, specific dates, specific market conditions where the strategy bleeds. The backtest is not a proof. It's a discipline. It forces you to define what "working" means before you're anchored to the live behaviour of a system.
That lesson transferred almost verbatim to working with large language models in production systems. Most teams I've seen reach for the model before they build the scaffold to judge it. The model gets integrated, it produces outputs that seem fine, and then some combination of a user complaint, a bad demo, or a regulatory question reveals that the team has no way to say with confidence what "fine" means or whether they're getting it consistently. The harness should have come first.
What a harness actually is
The word "harness" is overloaded in AI engineering the way "observability" was overloaded in backend engineering ten years ago — everyone agrees it's important, nobody agrees what it contains. In traditional software, a test harness is the scaffolding around a unit under test: inputs, expected outputs, assertions. The contract is explicit. An AI harness is different because the expected output is probabilistic and context-dependent. You're not asserting a value. You're asserting a range of acceptable behaviour across a distribution of inputs.
That difference is significant. It means the harness isn't just a runner — it's also a judgment system. You need to define what acceptable looks like before you can measure whether you're achieving it. For most backend engineers trained on deterministic systems, that shift is uncomfortable. The discomfort is exactly the right place to do the design work.
The signals worth capturing
Before you write a single eval, you need to decide what you're measuring. Latency and cost-per-call are operational signals. Hallucination rate on a held-out set, instruction-following on edge cases, and output degradation across model versions are quality signals. These are not the same thing, and conflating them in a single harness is where most frameworks go wrong.
The teams I've seen do this well pick two or three signals that map directly to user harm or business cost. Not the full list of what's technically measurable — the short list of what actually matters for their system. A summarisation model and a code-generation model have almost nothing in common in terms of the signals that matter. The harness should reflect that specificity, not abstract it away.
A shape that has worked
At a high level, an eval harness has four parts: a dataset of (input, expected_behaviour) pairs, a model runner, a judge, and a report. The dataset is the hardest part. The runner is usually three lines of code. The judge is where you make a series of uncomfortable choices about what "acceptable" means. The report is what you look at when something changes.
# A minimal eval harness structure
class EvalCase:
input: str
expected_behavior: str # prose description, not exact output
category: str # "routine" | "adversarial" | "regression" | "boundary"
severity: str # "low" | "high" | "critical"
class EvalResult:
case: EvalCase
model_output: str
judge_score: float # 0.0–1.0
judge_reasoning: str
latency_ms: int
cost_usd: float
class EvalReport:
run_id: str
model_version: str
timestamp: str
results: list[EvalResult]
pass_rate: float # judge_score >= threshold
p95_latency_ms: int
total_cost_usd: float
regressions: list[str] # cases that passed last run, failed this oneThe dataset needs adversarial cases — inputs that look routine but expose failure modes. Building that dataset is where the real engineering judgment lives, and it requires you to think like someone who wants the system to fail. This is not a job you can delegate to the team that built the system. They have blind spots. Bring in someone from outside, or build an adversarial dataset generator as a separate workstream.
The judge problem
Once you have a dataset, you need a judge. The options are: human labelers (expensive, slow, gold standard), rubric-based scoring (cheap, brittle at the edges), or a judge model (fast, scalable, comes with its own failure modes). In practice, most production systems end up using a judge model for routine regression runs and human labelers for the dataset itself and for cases where the judge model's score is ambiguous.
The risk with judge models is circularity. If the judge model was trained on similar data to the model under test, its judgments are not independent. It may rate confidently wrong outputs as acceptable because the failure mode is systematic across both models. The judge and the defendant should not share a training distribution if you can avoid it.
def run_eval(case: EvalCase, model, judge_model) -> EvalResult:
start = time.monotonic()
output = model.complete(case.input)
latency_ms = int((time.monotonic() - start) * 1000)
judgment = judge_model.complete(f"""
You are evaluating the output of an AI assistant.
Task description: {case.expected_behavior}
User input: {case.input}
Model output: {output}
Score the output from 0.0 (unacceptable) to 1.0 (fully acceptable).
Respond with JSON: {{"score": float, "reasoning": str}}
""")
parsed = json.loads(judgment)
return EvalResult(
case=case,
model_output=output,
judge_score=parsed["score"],
judge_reasoning=parsed["reasoning"],
latency_ms=latency_ms,
cost_usd=estimate_cost(case.input, output),
)When to build it
The temptation is to build the harness after the model is integrated, once you "know what you need." That's the same logic as writing tests after you've shipped the feature. It's also wrong for the same reason: the harness forces you to define what "working" means before you're anchored to the current model's behaviour.
Build the harness when you're writing the integration spec. The first version doesn't need a judge model. It needs a dataset of twenty to thirty cases — a mix of routine, adversarial, and boundary — and a rubric that a human can apply in thirty seconds per case. Automate the judge later. Start with the discipline of having to make a judgment at all.
The trading analogy holds. You don't commit capital without a backtest. Don't commit users without a harness.