The mandate that came back bigger than the ask
I pitched a four-part internal course on disciplined engineering practice in an AI-accelerated codebase: development and branching, testing, deployment and versioning, planning and agile delivery. The firm approved it. What came back with the approval was a bigger brief than the one I pitched — fold the course into a wider initiative, roll the practices out across every project, measure the improvement within the following week, and keep tracking it from then on. I was made personally answerable for whether that measurement holds up.
My recommendation, stated first because that is how I'd want to read this if someone else had written it: take the course, decline the one-week measurement window, and negotiate a longer one before committing to any number. The rest of this is the evidence for why.
Why a one-week window can't answer the question being asked
The course itself is straightforward. Session one is development and branching — prompting with intent, reading every diff hunk before it merges, keeping changes minimal, never pushing code you can't explain in your own words. Session two is testing, framed as measurement before automation. Sessions three and four cover deployment and versioning, then planning and agile delivery. None of that is controversial, and teaching it is the easy half of the mandate.
The hard half is proving it worked, on a clock that is structurally too short for the thing being asked. Two categories of signal exist here, and they move on two different clocks.
- Compliance signals — did people start naming branches correctly, did push cadence change, did the number of diff hunks reviewed before merge go up. These are visible within days. They are also, on their own, silent about whether anything actually got better.
- Outcome signals — did time-to-merge improve, did the rework rate drop, did the number of defects that reach production go down. These need weeks, sometimes longer, because they are noisy at the scale of a handful of days and only settle once enough volume has passed through the system to average out the noise.
A five-day count of the first category will always exist. A five-day count of the second category will always be too thin to trust, no matter how carefully it's collected — the sample simply isn't big enough yet for the signal to separate from the noise. Reporting the first as if it answered the second is the trap, and it is an easy trap to fall into precisely because the compliance number is the one that's ready on time.
Two tiers, reported separately, never substituted
The discipline I've settled on is not clever. It is keeping the two tiers explicitly labeled, in every single report, rather than letting a strong compliance number imply an outcome claim it hasn't earned yet. "Adoption is up" is a true and useful sentence. It is not an answer to "is the practice working," and treating it as one will eventually meet a reader who asks the outcome question directly. At that point, having no real answer costs more credibility than being early and honest would have cost.
There is a second reason to keep the numbers apart, beyond honesty: measuring adoption in week one changes the behaviour you're measuring. People branch correctly and review diffs more carefully when they know someone is counting, for exactly as long as they know someone is counting. That effect is real, it is not the same thing as the practice becoming habitual, and a one-week compliance spike that's driven by being watched will not survive the observation ending. Any early number has to be read against that possibility, not reported as if the observer weren't part of the system being measured.
What a defensible early signal actually looks like
Given a one-week window, I don't try to manufacture an outcome number early. I look backward before I look forward. The firm already has a history — closed tickets, merge timestamps, defect reports from before the course ever ran. Mining that retrospective record for a baseline, even a rough one, is worth more than a week of freshly collected outcome data, because the baseline doesn't carry the observer effect the new data does, and it gives the eventual outcome numbers something honest to be compared against.
So the week-one report I'd actually stand behind has three parts: the compliance numbers, labeled as compliance; a stated baseline pulled from history, labeled as a baseline and not a result; and an explicit statement of when the outcome numbers will be defensible — not "soon," a real date, tied to how much volume needs to pass through the system before the noise settles.
Why being personally answerable changes how carefully you build it
Being the one who has to stand behind the measurement, not just deliver the training, changes what counts as done. It's one thing to teach a practice and hope it sticks. It's another to be the person who put a number in front of leadership and has to defend it later against whatever the real trend turns out to be. That accountability is exactly what makes the two-tier discipline non-negotiable rather than a nice-to-have — a measurement story that collapses compliance into outcome is a story I would eventually have to walk back myself, in front of the same people who approved it.
What I do now
- I label every metric in a rollout report as either compliance or outcome, in the report itself, never letting one stand in for the other.
- I pull a retrospective baseline from existing history before reporting any new number, so the first real measurement has something honest to be compared against.
- I state, in writing, how many weeks an outcome signal needs before it's trustworthy — before the first week starts, not after the first number looks good or bad.
- I flag the observer effect explicitly in any early compliance number, rather than letting a strong week-one result imply more durability than it's earned.
- I say "not yet measurable" when that's true, in the same tone I'd use to report a good number, because a clean-sounding early result and an unexamined one look identical from the outside.