The Testing Spine

At the September 4 snapshot the factory’s fast suite holds 5,340 cases and runs green on every merge. I’ve stopped finding that number interesting. A large green count tells you what the suite asserts; it tells you nothing about what it finds. The question worth publishing is narrower: for each rung of the verification stack, which defects did it catch, what did the rung cost, and what could it not see by construction?

That question became answerable on August 22, and not before. On that day every factory session started carrying two labels — the phase it was working (plan, dev, qa, review, later retro and prep) and the story — and every finding started carrying a found-by: marker naming the instrument that caught it. Nothing was backfilled. Stories that merged before that date have one cost lane and no attribution; stories after have a per-phase bill and a per-instrument tally. The discontinuity is visible in every table below, on purpose: the instrumentation boundary is a finding, not a footnote.

(Series rules apply: running totals, never final ones, every number dated and regenerable — and where the instrument doesn’t exist, the tables say not measured rather than filling the cell.)

The Rungs

Seven rungs, from the cheapest to the one only a human can run. Post 3 described the oracle gate; the rest grew around it.

Three things about that table matter more than its rows. The browser rung went from 26 cases to 87 in eight days — the HUD stories — which is a 3.3× jump in the one rung that can see a z-index outranked by its own file’s wildcard, something no jsdom test will ever report. The E2E lane is still local-only, because CI would first need a measured way to send an access-gateway’s headers on a WebSocket upgrade, and its median has grown to nearly seven minutes as the suite widened. And the mutation rung — the one post 4 argued is the only defense against a green that proves nothing — pushes no telemetry at all. Its counts exist as prose in story files, and this post quotes them as prose. A rung with no event stream cannot answer “how often,” and I’d rather publish that gap than a plausible number.

Who Found What

Here is the first table that couldn’t exist before August 22: for each Epic 6 story, findings attributed to the instrument that caught them, exactly as the factory’s own counting script prints it at the September 4 tip.

Read it honestly and three things stand out. The walk ? is deliberate and unchanged since the first delta: walk findings are prose, present but uncounted, and a number nobody counted would be worse than a question mark. Review dominates every row, which is real — the machine-read review tallies for these stories run from 31 to 60 findings each. And the six zeros in the qa-agent column are wrong, which is the next section.

But first the row that reframes the whole table. Story 6.15 ran a fourth review layer alongside the three standard ones — a reviewer that asks only one question of each guard the diff touched: what mutation turns this assertion red? On a diff where 3,200 of 7,734 added lines were test code, that layer found more real defects than the other three combined. Gutting five methods of a shipped storage vault left 36 of 36 tests green. Two guards were vacuous, one of them passing on a ReferenceError because a function was never imported and the test had never run its subject at all. And the review’s own fix delta contained a regression — a prefetch reordering that silently made the character’s own tile the oldest of its group — caught only because a mutation happened to be run against it. The best-yielding instrument on the most test-heavy story of the epic was the one that assumes the tests are lying.

The Instrument That Counts Instruments

The per-instrument table is produced by a script, and the script’s QA-agent column is fed by a regular expression that matches headings of the form ### QA-. That is the heading style of exactly one QA report: the first one, for story 6.2. Every report since — five stories — used a different convention: numbered findings, numbered walk steps, per-story letters. So the counter printed qa-agent 0 for five consecutive stories that each ran a QA agent and each filed findings. The real counts, read from the found-by: qa-agent markers the reports themselves carry:

And the one row the counter did populate, 6.2’s 15, is an upper bound: the script adds report headings to ledger entries without de-duplicating, and at least three findings appear in both. Nine distinct QA findings is the defensible number.

The defect was found on September 4 by a session in the product repo answering questions for this series — while assembling the table above — and it’s recorded as a correction in the dated delta rather than fixed in the same commit, because a correction owned in print and a code change are different acts. What makes it worth a section rather than a footnote is its shape. The whole point of attributing findings to instruments was to publish the tester-versus-developer comparison. The instrument computing that comparison had been under-reporting the exact lane the comparison is about, for five stories, while every one of those reports carried a machine-readable tag the counter never read. The behavior was correct and the instrument was lying — the sentence that has run through this series since the first cost dashboard under-counted by 2.6× — this time about the instrument that measures instruments.

Two smaller findings from the same session belong here. The project’s global findings ledger, its longest-running artifact, quietly stopped on August 26; findings since live in story files under a per-story letter scheme, and nothing announced the change. And one ledger id is used twice for two different findings. Neither is dramatic. Both are the kind of drift that a counter reading zero will happily conceal.

What a Story Costs, by Phase

The other table that couldn’t exist before the boundary: per-phase cost for every story that merged with labels on. All figures at API list rates (the cash caveat applies: the work ran on a subscription, roughly 10× cheaper), from the phase-attributed query pinned at 2026-09-04T21:00Z.

What the split shows, on ten stories — and ten is few, so read these as observations rather than laws:

• Planning is not small. On the combat-domain story, planning cost more than development: a codec story is decided at creation. On the UI stories the ratio inverts. Nothing about the old single-lane cost could have told me which.
• Review is the widest-variance phase, $83 to $507, and on three stories it cost more than development did. Post 7 makes the case that it is also the least instrumented.
• The QA agent is 4% to 12% of a story, and it ran on eight of ten. The two it skipped were a ruling (a codec story with no surface to play) and a story with nothing for an agent to walk. That’s the design: the QA lane is selective, not sequential. It hunts for surprises before a human’s device walk, which is expensive and consequential; the E2E lane protects scripted regressions. Running the agent on every story would spend the budget where the walk already looks.

And the cumulative view since the labels arrived: development $3,275, review $2,660, planning $1,352, QA $433, epic prep $145, retrospective $82. Two of those phases were invisible before — prep work used to vanish into the planning bucket, which the retrospective that added the label described as a measurement scheme’s own categories are a place things hide.

What Each Rung Cannot See

The useful way to read a testing stack is by its blind spots, because the blind spots decide where the next dollar goes.

L0 cannot see the browser. A plate rendered in the wrong half of the screen survived 3,583 fast tests, 24 browser-rung tests, a clean typecheck and a passing E2E run; a screenshot found it. That’s why the browser rung tripled during the HUD stories, and it’s why the device walk exists.

Review cannot see execution. Post 4 put it as reading versus driving, and the score hasn’t moved: every vacuous guard in Epic 6 — three on one story, four on another, nine on a third, two on the most recent — was found by a mutation, none by a reader. Review is the highest-yielding rung by count and structurally blind to the defect family that matters most in agent-written tests.

The QA agent cannot see the phone. It plays the built bundle in a desktop browser at a phone viewport, which finds copy, flow and state defects — and it found a way to settle “is this really the realm’s ground” from the desktop that the human walk no longer has to. But the ground under a character on an actual phone, persistent storage prompts, install UI: those remain the walk’s, and the walk’s findings remain uncounted.

The oracle cannot see what the desktop client never sends. The deviation register’s twelfth entry records that the reference client refuses out-of-range and on-cooldown casts locally, so two whole result codes appear in no capture and can never be diffed. Thirteen approved deviations, zero pending, and the newest one carries the register’s first expiry date.

The counter cannot see a heading it wasn’t told about. Above.

The pattern: each rung was added when a measured failure showed the rung below it couldn’t have caught the defect. None was added because a pyramid diagram said it should exist. The E2E lane is the cautionary case in the other direction — seven minutes per run, local-only, and its four findings on one story are the only ones it has been credited with since attribution began. It may be earning its keep as regression insurance; the data can’t yet say. That’s what the next delta is for.

Regenerate This

The phase table is one query against the project’s metrics store, pinned to the snapshot’s evaluation time. max_over_time is load-bearing: a resumed session keeps its id and restarts its counter at zero, so last_over_time read one story’s $52 QA walk as $4.78 before the idiom was fixed.

sum by (story, phase) (
  max by (story, phase, session_id) (
    max_over_time(claude_code_cost_usage{project="pwa-wotlk", phase!=""}[90d])))

Run counts and medians per layer come from the test-run event series, count_over_time and quantile_over_time over thirty days, grouped by layer and trigger. Suite sizes read the per-kind case gauge at the last run. The per-instrument table is the attribution script’s Found by column, with its known regex defect; the corrected QA counts are a grep for found-by: qa-agent across the QA reports. All recipes ship in the product repo’s query file when it goes public, alongside the dated delta that this post is drawn from.

Final Thoughts

If you take one practice from this post, take the label. The day every session said which phase it was in and every finding said which instrument caught it, the factory’s testing stack stopped being a diagram and became a budget — one where I can see that review costs as much as development and finds the most, that the QA agent costs a tenth of a story and finds what nobody else does, and that the rung with the best yield on the hardest story has no meter at all.

Use test-layer metrics to decide the next verification investment. Never use them to celebrate activity. Five thousand green tests and a defect in the wrong half of the screen is not a contradiction; it’s a description of what green means.

The series keeps running with the factory. The hub carries the freshest totals, the register from post 7 carries what breaks next, and when the counter is fixed, this post gets a dated second reading — with the numbers it should have had.