My recommendation, then the cases it comes from: before trusting any single clean result, name the second measurement that would disagree with it if the first one were wrong — and run it once, before it matters, not after. Nearly every verification failure I've traced back to a root cause turns out to be this same shape wearing a different costume: one number, one test, one green check, taken as the whole answer when it was only ever one angle on it.
Drift is not quality
The clearest version of this happened during an A/B comparison of a model change for a client's self-hosted LLM deployment. I rejected the new configuration because thirty-three percent of its classifications differed from the current one — a large, alarming-looking shift. Then I went back and scored both configurations against the corpus's own ground-truth labels, which I hadn't done the first time. The rejected configuration was more accurate: 53.0 percent versus 47.7 percent for the one I'd kept. My rejection had been wrong on evidence I already had sitting two clicks away.
The first measurement — percentage of outputs that changed — answers a real question: how large is this behavioural shift. That's genuinely useful, because you want to know a change is big before you ship it. But it doesn't answer which direction the change went, and treating a large shift as automatically a bad one quietly assumes the current behaviour was correct to begin with, which is exactly the thing under test. Only labels answer direction. Size and direction are different measurements, and I'd taken the first for the second.
The part I find most useful in hindsight: before concluding that accuracy simply couldn't be measured and falling back to drift as the only available signal, check whether the dataset already carries labels. It often does. I once spent two days calling a measurement "impossible" on a corpus that had carried ground-truth labels the entire time, because I never ran the second query that would have shown me.
A guard tested by hand is a different guard from the one that ships
Verifying a check by running it yourself, from a terminal, on demand, proves almost nothing about how it behaves in the place it actually runs. I hit four separate versions of this in a single day auditing gates across the same system:
- A pre-push safety gate passed every time I ran it by hand and failed every time it actually ran as a real pre-push hook. The cause: the hook environment set two git variables that silently overrode the working directory the gate's own code thought it was checking, so a real, large divergence between branches was reported as zero.
- A command meant to detect drift between two checkouts reported everything fine — because the shared working copy it was reading from had been left on the wrong branch, so it was comparing the wrong tree to itself.
- A test's skip condition checked whether a database connection could be opened, not whether the data it needed could actually be read. Whether the test ran at all quietly depended on an unrelated command someone had happened to run earlier that day.
- A fix that had been merged, tested, and pushed kept producing bad output in production. The service running it pointed at a different checkout, nearly two hundred commits behind, whose own status command reported itself clean and current — because it was comparing against a frozen reference of its own, not the branch everyone else was working from.
The direction of failure in all four matters more than the fact of a failure. A check that fails closed by mistake wastes an afternoon. A check that fails open inside its own deployment context does something worse: it launders real, unfixed debt into a record that says the debt was cleared, and anything built on that record — a dashboard, a report to a client, a decision to stop worrying about it — inherits the same false floor. The way to catch this is not more testing on demand. It's exercising the guard exactly where it will actually run: from the real hook, the real timer, the real pinned checkout, not a clean terminal session that skips every piece of context the real invocation carries.
When the check that ran might not have run at all
There's a sharper version of the same trap, one level down, that shows up when you're testing the tests. I use mutation testing on some of these gates: deliberately introduce a known bug and confirm the test suite turns red. Twice in one day, a mutation that failed to apply — the code edit simply didn't land — produced the exact same terminal output as a mutation that applied correctly and the test suite failed to catch. Both read as "green when it should have been red." One meant the test had a real coverage gap. The other meant nothing had been tested at all, because the experiment itself never happened.
Once, the text I was matching against to make the edit had drifted since the harness was written, so the edit silently did nothing. Once, a second, chained edit no-opped and left behind a syntax error that a simple text search for "failed" never matched, because the test runner reports a syntax error as an error, not a failure. Both times my first instinct was to write "the test suite has a hole here." The honest read was "this experiment never ran," which is a different defect with a different fix. A mutation harness that can't tell its own absence from its own success is broken in exactly the way it exists to find in other people's code — so now mine has to confirm the patch actually changed the file, confirm the result still parses, and count an error the same as a failure, before it's allowed to report anything at all.
The protocol
The common thread across all of these — the model comparison, the four gates, the mutation harness — is that a single measurement, however carefully taken, only ever answers the question it was built to answer, and a clean result gets quietly read as answering a broader one. The fix isn't distrust of every result forever, which would make nothing shippable. It's a habit applied at the one moment it matters: before a green result gets acted on, name out loud what a second, different measurement of the same claim would look like, and run it once. If naming that second measurement is easy, run it before you trust the first one. If it's hard to even name, that difficulty is itself the signal that you don't yet know what the check actually covers.
What I do now
- Before trusting a drift number, I ask whether the corpus already has ground-truth labels, and I score both sides against them before calling anything a regression.
- Every guard I rely on gets exercised at least once the way it will actually run — from the hook, the timer, the pinned checkout — not just from a terminal where I control every variable it depends on.
- Any check that crosses a boundary — asking about a different repository, service, or checkout than the one it's running in — gets its environment stripped and re-checked explicitly, because inherited context is exactly where this kind of failure hides.
- Any mutation or fault-injection test asserts that the fault actually landed before it asserts anything about whether it was caught.
- When a result is clean and I didn't have to work for it, I treat that ease as a prompt to ask what a second measurement would show, not as a reason to move on faster.
Related: A Check Is Only As Honest As Its Name covers the specific case where the first measurement's own name overstates what it checked, and Right Number, Wrong Story covers the case where the first measurement was correct and the story built on top of it wasn't.