My recommendation: before you trust or critique any single parameter in an AI evaluation suite, ask one question of every parameter in it — does this measure whether the software behaved correctly, or whether the work it produced was any good? Then count. Suites drift overwhelmingly toward the first, because machine behaviour is cheap and deterministic to check, and output quality needs someone to apply domain judgement. The drift is invisible until you count it on purpose.
Nine parameters, zero of them about the work
I reviewed an eval suite once for a physics-informed AI product — nine parameters, all nine of them measuring machine behaviour. Did the router pick the right mode. Did the output parse. Did the state machine advance correctly. Not one of the nine measured whether the engineer using the tool actually got a valid, useful artifact out of it, which was the product's own stated definition of success.
The suite hadn't been built badly. It had calcified around whatever crisis created each parameter. Four of the nine existed because of a single physics-fidelity investigation months earlier — a real incident, a real fix, a real parameter added to make sure it didn't regress. The product's actual differentiator, the part that made it worth building at all, had zero coverage. Nobody had decided to leave it uncovered. It simply never produced a crisis, so nothing ever forced a parameter into existence for it.
That's the pattern worth naming: an eval suite's shape tells you the history of what broke, not the shape of what the product needs to be good at. A suite that's never seen a quality incident in its most important dimension will have no parameters there, and a green run across all nine will read as health. It isn't. It's silence in exactly the place nobody has ever had a reason to make noise.
The coverage-matrix check
The fix isn't a smarter metric. It's a matrix: product surface down one axis, what each parameter actually measures across the other, filled in before judging whether any individual number is good or bad. The gap doesn't show up as a bad metric — a bad metric at least invites scrutiny. It shows up as a column of zeros, and a column of zeros is invisible until you build the matrix on purpose, because nothing about a nine-parameter suite with nine green checkmarks looks incomplete from the outside.
Two things fall out of doing this seriously:
- Sequence gaps by lead time, not by dependency order. In the case above, a golden dataset needed to close the coverage gap sat third in the technical build order but depended on someone else's calendar to assemble — so it should have started first in wall-clock time, not third in dependency order. The gap you find in the matrix is only useful if you also fix the thing that makes closing it slow.
- A suite that measures only machine behaviour will always look done. Machine-behaviour checks are the ones that are cheapest to write, cheapest to keep green, and easiest to add "one more of" when something breaks. None of that effort ever closes the column that actually needed a person's judgement.
When the gate itself is the bug
A worked case makes the risk concrete. On a self-hosted LLM serving deployment, a benchmark gate reported the model failing five separate times over one investigation, each instance convincing enough on its own to act on:
- A hand-written JSON-repair step reported "43% of batches fail to parse." The production system's real parser, run on the same output, gave 30 out of 30.
- A grader that couldn't read positional arrays would have scored every one of them 0% schema-valid, for a shape the real consumer read without complaint.
- Calling
.upper()on a field the model had legitimately emitted as a bare integer threw an exception; the caller recorded the request as failed. Arms that were actually complying with the schema scored 4–6% parse rate against a control sitting at 99.3%. - A field the schema said was absent by design got flagged as missing.
- A process-watch check matched its own command line in a process list and waited forever for a process that was never going to exit, then reported the run as healthy.
Every one of these is the same failure shape: a gate encodes one contract, a different but equally valid output shape shows up, the gate fails, and the failure gets attributed to whatever it was pointed at rather than to the gate. Five instances, one investigation, each one plausible enough that acting on it without checking further would have thrown out a working model.
What actually caught these: porting the real parser instead of reimplementing it, and verifying the port by running the original parser against the same data before trusting the new harness's verdict at all. Passing the contract in explicitly from the corpus rather than letting the grader infer it. Reporting grader errors as grader errors, in their own field, so a broken gate can't quietly present itself as a quality result.
Drift is not quality
A related trap shows up the moment you A/B a model or a prompt change. "N% of outputs differ from the control" is drift, not degradation — the control is today's behaviour, not the correct answer, and reporting a change in behaviour as a loss in quality silently assumes today's output was right to begin with.
I rejected a serving-stack change once on exactly that basis — 33% of classifications changed from the control — and only afterward went back and scored both arms against the corpus's own ground-truth labels, which had been sitting there the whole time. The rejected arm was more accurate than the one I kept. My rejection was wrong on evidence I already had, two clicks away from the number I acted on instead.
Drift and accuracy are answers to different questions. Drift bounds how large a behavioural change is, which is genuinely useful — you want to know a change is big before you ship it. But only labels tell you which direction the change went, and treating the size of a change as a proxy for its direction is a category error dressed up as caution. Before declaring accuracy unmeasurable and falling back to drift as the only available signal, check whether the corpus already carries labels. It often does. I once spent two days calling accuracy "impossible to measure" on a dataset that had carried ground-truth labels the entire time.
What I do now
- Before critiquing or trusting any single eval parameter, build the coverage matrix — product surface by parameter count — and read the zeros before reading the scores.
- When a suite reports a failure, check whether the failure belongs to the thing under test or to the gate itself, by porting and re-running the real consumer's logic on the same output before accepting the grader's verdict.
- Report grader errors as grader errors, in a field of their own, so a broken harness can never present itself as a quality result.
- Treat "N% changed from the control" as a drift measurement, full stop, until a separate pass against ground-truth labels says which direction the change went.
- Check for existing labels in the corpus before declaring accuracy unmeasurable — it's a shorter check than building a new labeling pipeline, and it's often already sitting there.
The same discipline of reading a claim narrower than it's presented applies to any check whose name promises more than its code delivers — an eval suite is just that pattern at the scale of an entire product.