A Check Is Only As Honest As Its Name

Every automated check makes a promise before it runs a single line: its name. "No override path." "Verify the row exists." "Every event anchored to this ticket." Read as English, each of those is a claim about the world. The trouble is that a check is code, not English, and the two drift apart in a specific, boring, recurring way — the code answers a narrower question than the name promises, and nobody notices, because nobody re-reads a check that is passing.

My recommendation, before the evidence: read a check's name as a claim, and before you trust a green result, write out the query that would have to run for that claim to be honest. Then confirm the code in front of you actually runs it. This is a five-minute habit, not a new tool, and it is the single defect I keep finding most often — in systems I've built and in other people's.

Six gates, six gaps, one day

I run a personal system called Astram — an append-only event log that every project, decision, and status report I make derives from, so I have one place that says what actually happened rather than what I remember happening. It has a lot of gates: things that block a bad commit, refuse a stale checkout, confirm a background job actually ran before letting a report say it did.

In one day of auditing it — going gate by gate, exercising each one rather than reading its source and nodding — I found six of these in six different places:

1. A gate meant to confirm a checkout was current instead confirmed it matched my own local copy of history — a narrower, self-referential check dressed as a general one.
2. A safety gate meant to block every path that could bypass a review requirement blocked only the specific override flags I had thought to search for. A flag I hadn't pattern-matched walked straight through.
3. A lock meant to confirm a resource was held by another active session actually confirmed that some session ID was recorded somewhere, with nothing checking whether that session's lease had already expired and nothing had ever released it.
4. An install check meant to confirm a dependency had actually installed only confirmed that a folder with the right name existed on disk.
5. A query meant to select every live event anchored to a work item instead selected every event ever written against it, including ones that had since been formally retracted.
6. A function literally named "verify the row exists" verified nothing at all. A broad exception handler wrapped around it swallowed a bad API call before the check could run even once, and every caller read the silence as success.

Two of the six had a comment sitting directly above the code, stating the correct behaviour in plain language. The comment was right. The code next to it was narrower. That detail is the one that stays with me — the person who wrote both knew what the check should do and didn't get there, and nothing in the system's daily green output ever said so.

Why the gap survives

The reason this kind of gap survives for weeks instead of hours isn't that anyone involved didn't know better. It's that a green result is read as "the thing I named is true," not as "the specific query I wrote today returned no matches." Those are different claims, and only the second one actually happened. The first is what everyone assumes, including me, right up until something forces the gate to run for real.

That's also why exercising a check catches this and reading it usually doesn't. Reading a check, I bring the same assumption the name invites. Exercising it — planting a case the name says should fail, and watching whether it actually does — has no assumption to bring. All six of the gaps above were found the second way. None were found the first.

The same blindness shows up one level earlier, in the audits that decide whether a check is even needed. I once concluded, from a name-based search across a codebase, that roughly forty percent of a set of components had no usage and were candidates for retirement. Two of the ones on that list were not dead at all — they were live and heavily used, they just didn't log under their own name, so a search for their name found nothing. A two-in-seventeen error rate on a "these are unused" finding is high enough to invalidate the whole list, not just caveat it. The fix was the same discipline in a different direction: judge a component by what it declares — the event types it emits, the table it writes to — never by grepping for its name.

Two corollaries

A broad exception handler is worse than no check at all, because it converts a programming error into a silent no-op that looks healthy. If a guard's job is to fail on a real condition, it should fail open only on the thing it was actually built to tolerate — a network timeout, a flaky connection — never on a bug in itself. I now write every exception clause around a guard naming the specific failure it forgives, not a bare catch-all that will happily swallow the next coding mistake too.

The second corollary cost me more, once. Proving that a piece of code doesn't exist by running a search and reading the first page of results is not proof. It's reconnaissance. I once piped a code search through a line limiter, twice, and concluded a feature had never been built. It had — implemented further down the same output I'd truncated away, a whole router already doing the job I was about to plan from scratch. The correction rewrote weeks of planned build-from-nothing work into days of wiring up what already existed. Count the results before declaring an absence. Read all of them, not the first page.

What I do now

• Before trusting a passing check, I write out in one sentence the query that would have to run for its name to be an honest claim, then confirm the code actually runs that query.
• Any exception handler wrapping a guard names the specific failure it forgives. A bare catch-all around a safety check is a defect on sight now, not a style choice.
• Any search whose conclusion will be "this doesn't exist" gets counted before I read a single result. If the count runs past a page, I read the rest before I write that sentence.
• When I judge whether a component is used, I search for what it declares — its outputs, its event types, its written rows — never its name.
• A comment that states correct behaviour next to code that doesn't do it is now a specific red flag for me, not a reassurance. The author already knew the right answer and the code still doesn't have it.

Related: Right Number, Wrong Story applies the same discipline to conclusions built on a correct count, and The Discipline of the Second Measurement generalises it past named checks to any single clean result.