Failing in the Safe Direction
On September 21 the session that maintains my factory’s measurements committed its seventh dated facts file: careful, caveated, every number tagged with how to regenerate it. I read it, checked its file-derived counts against the repository myself — all six reproduced exactly — and then sent the session 25 questions for the posts I was about to write.
It answered all 25. Eight of the answers were corrections to its own file. Four of them changed numbers that were hours from being published, one of them on this series’ front page.
This post is about the pattern in those eight, because I’d seen it before and hadn’t named it. Nearly every measurement error this factory has made has under-reported. Lower cost. Fewer findings. One fewer hit. A quiet week. That isn’t luck, and it isn’t specific to agents.
(Series rules apply: running totals, never final ones; cost figures are metered at API list rates, with real cash roughly 10× lower.)
The Catalog
Here is every instrument in this series that turned out to be wrong in a countable way, expressed as one number each: what it reported, as a share of what was actually there.
The first bar is post 1’s: a cost dashboard that read 2.6× low because counters reset per session and nothing labelled the sessions. The next two are post 8’s: a counter that printed zero QA findings for five stories that had filed 27, and then, in the same forty-line function, lost a third of an epic’s review findings.
The last four arrived together, in one interview.
Four in One Afternoon
The cost query assumed one model per session. The recipe took the maximum per session, which is correct for a counter that restarts — and silently wrong for a session that switches models mid-run, because each model is its own series and the maximum keeps one. Planning sessions switch models. So planning read 8.4% low, the epic $182 low, and because two earlier snapshots used the same recipe, the series’ cumulative total read $183 low. The session found it while answering my questions, one of which — why is review so expensive? — needed cost broken down by model. Planning turned out to be the only phase with a model mix, and cost-by-model summed to more than cost.
The hand count made the mistake the file was documenting. The facts file devotes a section to a parser that missed findings because a prose convention changed. In the same file, the device-walk tally — counted by hand, from each story’s close-out line — read eleven of fifteen. It’s twelve. The missed story recorded its walk findings under a differently worded heading. The session’s own sentence: “My hand-count made the exact mistake the delta records twice: a count keyed on a prose convention.”
A comparison was short, on no basis at all. One story’s review “cost more than eight complete stories”. By the file’s own ranking table it’s eleven. I asked which basis gives eight. “I can’t reconstruct how I got eight — it is a slip. Please quote the correction.”
And one instrument was declared healthy on paper. Every finding was supposed to carry a tag naming the instrument that caught it. The register item for that convention was closed as done in mid-September. But the convention had only ever been defined for the global findings ledger, the ledger quietly stopped taking story findings in late August, and so the tag — outside the QA agent’s own reports — exists on exactly two stories out of nineteen. A done-when met on paper, over an instrument that had stopped measuring.
How this number is measured
Six were listed up front by the interviewed session before its answers; two more came from follow-up questions. The reply is preserved verbatim beside the facts file it corrects, and the facts file gets a correction commit, not a rewrite.
How this number is measured
Checked independently with the facts file’s own regeneration commands before the interview. The numbers that were wrong were never the ones with a one-line regeneration recipe; they were the ones with an assumption inside the recipe.
How this number is measured
Both recipes re-run independently at the pinned evaluation time; both reproduce to the cent. 0.8% of the total, 8.4% of one phase. Small — and at least the fourth time in this series a published cost figure has been corrected upward. I went looking for one that was corrected downward and couldn’t find it.
Two of the eight didn’t under-report, and I’m not going to bend them to fit. The file called the end-to-end test lane “46% inert” because 38 of 82 rows were skipped; in fact the 82 rows are 41 tests run in two phone orientations, and half the skips are a portrait test correctly declining to run in landscape. The real figure is 9 of 41 — and it hides something worse than the headline: five combat tests skip because of the shared test character’s state, so the lane’s combat coverage mostly didn’t execute. And the per-layer “median durations” I had already written into a draft turned out to be the duration of one arbitrary run each, because every run is its own one-sample series and a quantile over one sample is that sample.
And there is one honest over-report in the catalog, which the chart above steps around by counting only the five stories that read zero: on the single story where that QA counter did match, it printed 15 findings for 9 distinct ones, because it added two lists without de-duplicating. One over-count against seven under-counts. I’d rather show the exception than claim a law.
Different directions, same mechanism. Each recipe was right about the shape it was written for, and silent when the shape moved.
Why It’s Always Low
Two reasons, and neither is about carelessness. The session that made these errors is the most careful author in this series: it says not measured where nothing exists, it pins its queries to a timestamp, and it corrects itself in print without being asked.
Parsers fail by not matching. A counter built on a pattern — a regex over headings, a grouping over labels, a grep over a close-out line — has one overwhelmingly likely failure mode: the world changes shape and the pattern stops matching. Non-matches are silent, and they subtract. It takes a fairly exotic bug to make a regex count things that aren’t there. It takes two asterisks to make it count nothing.
And the errors that inflate don’t survive. A bill that doubles overnight gets investigated by lunch. A findings count that triples gets a raised eyebrow in the next review. A counter that reads zero looks like a quiet week; a cost that reads 8% low looks like efficiency. Over-reports are self-announcing and get fixed; under-reports are comfortable and get kept. Run that filter for a few months and what’s left in your dashboards is a population of errors that all flatter you.
That’s why I’ve stopped reading “our numbers have been stable” as reassurance. Ask instead: when was one of our metrics last corrected upward? If the answer is never, you’re probably not accurate. You’re probably not looking.
The Cheap Defenses
None of these needs a platform team. Each one comes from a specific catch above.
Six defenses, and the failure each one would have caught
| Defense | What it looks like | What it would have caught |
|---|---|---|
| Zero is a claim | A counter whose input exists and whose count is zero fails loudly instead of printing 0. | Five stories of qa-agent 0 with a QA report sitting on disk; six stories of review findings reading nothing. |
| Check the cardinality your grouping assumes | Before aggregating, count the series per group. If you assume one and find two, the recipe is wrong. | The cost query: one assertion — max series per session is 1 — fails the day a session switches models. |
| Two independent bases for anything you publish | Derive the same number two unrelated ways and make them agree. | The story count is checked against both the tracker's done rows and the merged branch ids; that pair caught an off-by-one in September. |
| Make the parts sum to the whole | Any breakdown is also a checksum. | The model-collapse bug is one subtraction away: cost by model sums to more than cost by session. |
| Stamp, don't remember | A label written by a launcher script holds. A convention a session must remember to follow decays. | Phase labels, stamped by script, held for nineteen stories — the single gap is a session started around the launcher. Finding tags, written by hand, held for two. |
| Have an outsider ask for the basis | Someone who didn't build the number asks which query, which rows, which denominator? | "Which basis gives eight?" has no answer, and the absence is the finding. |
The first one stings a little. Post 3 is proud of a rule in the verification harness: comparing nothing is an error, not a pass — an empty case list exits non-zero, because an oracle that tested nothing proved nothing. The factory applied that rule to its gates in its first week and never applied it to its counters. A findings counter that prints zero for a story with a findings report on disk is comparing nothing, and it passed for weeks.
The last one is the one I’d underline. The interview didn’t work because the questions were clever. It worked because answering them meant re-deriving instead of re-reading — and the session doing the re-deriving treated each mismatch as a finding rather than an embarrassment. Post 7 described the first such interview as a sourcing method. This one made the case that it’s a verification lane: a number that has only ever been read by the process that produced it has never been reviewed at all.
What Held
It would be easy to read this post as “the numbers can’t be trusted,” so, plainly: every shape claim in the series survived. Development is still the largest phase. Planning plus review still out-cost building — the ratio moved from 1.25× to 1.28×, in the stronger direction. The test pyramid still has almost no middle. Every count with a one-line regeneration recipe reproduced on the first try.
What moved were magnitudes, by single-digit percentages, and they moved because the system is built to let them: numbers pinned to a timestamp, recipes printed next to results, a rule that a superseded sentence gets quoted rather than deleted, and an author — human or not — that answers “how do you know?” by going and checking. The corrections are not evidence the measurement is bad. They’re what measurement looks like when it’s alive.
Regenerate This
The bar chart’s seven ratios: 1 ÷ 2.6 from post 1’s reset-aware re-aggregation; 0 of 27 and 425 of 636 from post 8’s two counter regressions (the second is one Python line comparing both patterns over the epic’s story files); 8 of 11, 11 of 12, $2,168 of $2,350 and $23,478 of $23,661 from the September 21 interview. The two cost recipes differ by one word — model in the inner grouping — and both are printed in post 9. The interview reply is a file; it ships beside the facts files when the product repo goes public, corrections and all.
Final Thoughts
I started this series with a dashboard that under-counted by 2.6× while looking perfectly healthy, and wrote that the measurement layer needs the same adversarial treatment as the code. Four months later the measurement layer is far better — and it just produced four more under-counts in an afternoon, each in a recipe that had been correct when it was written.
So the lesson isn’t “build better instruments.” It’s that instruments are code, conventions are interfaces, and both drift. The defense is the same one the factory uses on its tests: don’t trust a green you’ve never seen go red, and don’t trust a number nobody outside the pipeline has ever asked you to re-derive.
If you take one question back to your own dashboards, take this one: when did one of them last correct itself upward?
