The Testing Spine

At the September 21 snapshot the factory’s fast suite holds 6,741 cases and runs green on every merge. I’ve stopped finding that number interesting. A large green count tells you what the suite asserts; it tells you nothing about what it finds. The question worth publishing is narrower: for each rung of the verification stack, which defects did it catch, what did the rung cost, and what could it not see by construction?

That question became answerable on August 22, and not before. On that day every factory session started carrying two labels — the phase it was working (plan, dev, qa, review, later retro and prep) and the story — and every finding was meant to start carrying a found-by: marker naming the instrument that caught it. Nothing was backfilled. Stories that merged before that date have one cost lane and no attribution. The discontinuity is visible in every table below, on purpose: the instrumentation boundary is a finding, not a footnote.

The two halves of that change aged very differently, and the difference is this post’s first result. The phase labels are stamped by a launcher script, and they held for every story since. The finding markers are written by hand, and outside the QA agent’s own reports they survive on exactly two stories. A first draft of this paragraph said every finding started carrying a marker; that was the plan, not the record.

(Series rules apply: running totals, never final ones, every number dated and regenerable — and where the instrument doesn’t exist, the tables say not measured rather than filling the cell.)

The Rungs

Seven rungs, from the cheapest to the one only a human can run. Post 3 described the oracle gate; the rest grew around it.

Three things about that table matter more than its rows. The browser rung went from 26 cases to 87 to 167 across three snapshots — a 6.4× rise while the suite around it grew 1.5× — and it is the one rung that can see a z-index outranked by its own file’s wildcard, something no jsdom test will ever report. The E2E lane is still local-only, because CI would first need a measured way to send an access-gateway’s headers on a WebSocket upgrade; a run takes a median of nearly six minutes and a 95th percentile of eight and a half. And the mutation rung — the one post 4 argued is the only defense against a green that proves nothing — pushes no telemetry at all. Its counts exist as prose in story files, and this post quotes them as prose. A rung with no event stream cannot answer “how often,” and I’d rather publish that gap than a plausible number.

Is It a Pyramid?

I called this post a spine rather than a pyramid because nothing in the stack was designed to a shape. But the shape is measurable, so here it is.

One trap first. The browser rung is inside L0, not beside it: the same 167 tests appear under two metric names because they run in the same Vitest invocation, on L0’s clock. Quoting “L0 6,741 · browser 167 · E2E 82” as three tiers counts the rung twice. The honest tiers are L0 without the rung, the rung, and E2E.

It is a pyramid — 160 : 4 : 1 — and it is a pyramid with almost no middle. The base is enormous and fast. The middle tier is 2.5% of the base. And the tip is thinner than its row count suggests. The E2E report shows 82 rows with 38 skipped, and the factory’s own facts file first read that as 46% inert. It isn’t: the 82 rows are 41 tests run in two phone orientations, and 19 of the skips are a landscape test correctly declining to run in portrait, or the reverse. The honest figure is that 9 of 41 tests ran in no orientation at all — two are operator-only captures, two are parked, and five are combat tests that skipped because the shared test character wasn’t in the state they need: not enough mana, no consumable seeded, a target already held. That’s the sharper finding. In the last recorded run, most of the lane’s combat coverage didn’t execute, and nothing went red to say so.

I don’t think the base is the problem. The middle is, because the middle is the tier that catches exactly the class of defect Epic 6 kept shipping: a CSS rule that is dead because a wildcard outranks it, a gesture the browser claims as a pan before the app ever sees it, a layout that survives every jsdom assertion because jsdom does no layout. Every one of those was found by a browser, a screenshot or a thumb. None was found by a unit test, and there were 6,574 of those.

The encouraging half: the middle is the fastest-growing tier in the building, and it grew because measured failures demanded it, not because a diagram did. It’s closing — from very far behind.

Who Found What

Here is the first table that couldn’t exist before August 22: for each Epic 6 story merged by then, findings attributed to the instrument that caught them, exactly as the factory’s own counting script printed it at the September 4 tip. I’m leaving it at that date on purpose, because what happened to it afterwards is the story.

Read it honestly and three things stand out. The walk ? is deliberate and unchanged since the first delta: walk findings are prose, present but uncounted, and a number nobody counted would be worse than a question mark. Review dominates every row, which is real — the machine-read review tallies for these stories run from 31 to 60 findings each. And the six zeros in the qa-agent column are wrong, which is the next section.

Look at which rows are rich. Only the first two stories name a unit test, an E2E spec, a mutation or a seam as the thing that caught something; every later row is review, the QA agent and a question mark. I first read that as a convention decaying. The interview for this post found something more structural, and the structural part is measured: the found-by: convention was defined as a rule about the project’s global findings ledger, the counting script reads non-QA markers from ledger lines and nowhere else, and the ledger stopped taking story findings on August 26, right after the second story. Findings moved into per-story records, where no such convention exists. So the column for unit, browser, E2E, mutation and seam could only ever be fed by one file, and that file went quiet. Why stories stopped filing to the ledger is inferred, not measured. What isn’t inferred is the postscript: the register item for “a found-by marker on every finding” was closed as done at a mid-September sweep — a done-when met on paper, over an instrument that by then measured one rung out of seven.

But first the row that reframes the whole table. Story 6.15 ran a fourth review layer alongside the three standard ones — a reviewer that asks only one question of each guard the diff touched: what mutation turns this assertion red? On a diff where 3,200 of 7,734 added lines were test code, that layer found more real defects than the other three combined. Gutting five methods of a shipped storage vault left 36 of 36 tests green. Two guards were vacuous, one of them passing on a ReferenceError because a function was never imported and the test had never run its subject at all. And the review’s own fix delta contained a regression — a prefetch reordering that silently made the character’s own tile the oldest of its group — caught only because a mutation happened to be run against it. The best-yielding instrument on the most test-heavy story of the epic was the one that assumes the tests are lying.

The Instrument That Counts Instruments

The per-instrument table is produced by a script, and the script’s QA-agent column is fed by a regular expression that matches headings of the form ### QA-. That is the heading style of exactly one QA report: the first one, for story 6.2. Every report since — five stories — used a different convention: numbered findings, numbered walk steps, per-story letters. So the counter printed qa-agent 0 for five consecutive stories that each ran a QA agent and each filed findings. The real counts, read from the found-by: qa-agent markers the reports themselves carry:

And the one row the counter did populate, 6.2’s 15, is an upper bound: the script adds report headings to ledger entries without de-duplicating, and at least three findings appear in both. Nine distinct QA findings is the defensible number.

The defect was found on September 4 by a session in the product repo answering questions for this series — while assembling the table above — and it’s recorded as a correction in the dated delta rather than fixed in the same commit, because a correction owned in print and a code change are different acts. What makes it worth a section rather than a footnote is its shape. The whole point of attributing findings to instruments was to publish the tester-versus-developer comparison. The instrument computing that comparison had been under-reporting the exact lane the comparison is about, for five stories, while every one of those reports carried a machine-readable tag the counter never read. The behavior was correct and the instrument was lying — the sentence that has run through this series since the first cost dashboard under-counted by 2.6× — this time about the instrument that measures instruments.

Seventeen days later the same forty-line function turned out to hold a second, larger hole. The review column is fed by a pattern that expects the tag [Review] to follow a checkbox immediately. From one story onward, six story files started writing that tag in bold, and two asterisks are enough to break the match:

So for six of the epic’s nineteen stories, the factory’s own attribution table currently prints a question mark for the walk and nothing else. Both regressions are recorded and neither is fixed yet: the first is an open item in the factory register, and I would rather publish that than pretend the promise at the end of this post’s first draft — “when the counter is fixed” — had been kept. The general lesson is the one this project keeps paying for: a parser keyed on a prose convention decays the moment the prose does — silently, and in the safe-looking direction. A counter that over-reports gets noticed. One that reports zero looks like a quiet week.

Two smaller findings from the first session belong here. The project’s global findings ledger, its longest-running artifact, quietly stopped on August 26; findings since live in story files under a per-story letter scheme, and nothing announced the change. (By September 21 it had taken exactly one more entry across eleven merged stories, while those stories filed 363 review findings and 51 QA findings into their own files. Not abandoned — superseded in practice, and nobody wrote that down.) And one ledger id is used twice for two different findings. Neither is dramatic. Both are the kind of drift that a counter reading zero will happily conceal.

The Rung That Runs Last

When Epic 6 closed on September 21, I asked for the per-rung yield across all nineteen stories. What came back is the most useful table in this post, mostly for its last column.

(Review’s parts sum to 687 against a stated 682 because stories differ on whether “dismissed” sits inside the total. Both are printed as the stories wrote them rather than silently reconciled.)

Only two instruments have a complete record for the epic, and that is itself a finding. But the third row is the one I want to read carefully, because the factory’s own facts file read it wrong twice and I nearly published both errors.

The first error was the count. The file said eleven walks of fifteen found something. It’s twelve: one story recorded its walk findings under a heading worded differently from the others, and a hand count keyed on the usual wording missed it — the same class of mistake as the two regex defects above, made by hand, in the file that documents them. The second error was the conclusion. The file called the walk the instrument with the best hit rate. It isn’t. At story level the QA agent found something on 14 of its 14 runs and code review on 19 of 19; the walk’s 12 of 15 is the lowest of the three. The session that wrote the sentence withdrew it when I asked whether it would stand behind it.

What’s true is stronger than what was claimed. The walk runs last. By the time I pick up the phone, a story has passed the fast suite, the browser rung, typecheck, the oracle gate, a multi-layer code review and, usually, a QA agent playing the build. And in twelve stories of fifteen, the walk still found something every one of them had passed over. Its yield is the residual, not the rate:

• A gesture that could not work on a phone at all. The holster is hold-then-slide, and the key never declared touch-action, so a real browser claimed the slide as a page pan and cancelled the pointer: the fan opened and vanished under the thumb. Every lane was green, each for its own structural reason — hand-dispatched pointer events never enter gesture arbitration, Playwright’s mouse is a mouse and touch-action doesn’t govern mice, and emulated touch has no compositor. The story file’s summary: one thumb, one gesture, one line of CSS.
• A quest giver ignored for fourteen taps, on the story whose entire purpose was proving the daily loop. The client understood a giver who offers one quest directly and silently discarded the chat-menu shape. Everything was green because, in the file’s words, the one quest giver the CI harness depends on takes the only route the client supported. The sting: the app’s own latency telemetry had recorded it anyway — 33 abandoned greets against 3 acknowledged — and nothing was watching that ratio.
• A spirit healer that worked, when every automated lane and the QA run had it silent. The walk doesn’t only find defects; it overturns false negatives.
• A mount-key tap that locked the player out of every ability, which became a deviation-register entry that lived two weeks.

The three walks that found nothing said so in as many words — one close-out reads “Nothing — and that is the measurement.” The most instructive nil is the character-creation story: nothing broke, and the walk still discharged a criterion no lane could reach, because the QA agent had stated its own limit precisely: I did not enter the world on a newly created character. An instrument that names what it didn’t do is worth more than one that reports green.

The walk is also the only instrument with no counter, no time series and no dashboard panel. Its column reads ? by a deliberate choice — a number nobody counted is worse than a question mark — and “12 of 15” is labelled as hand-derived everywhere it appears, which is exactly why it has already been wrong once. Nobody has proposed a walk counter and nobody has ruled against one. I’m not sure I want one; I am sure that the rung that catches what all the others miss is run by the human and is invisible to the factory’s telemetry, and that a budget conversation that only sees the dashboards will cut it first.

Two honest asterisks on the rungs below it. The browser rung grew 6.4× and I can find no recorded defect it was the first to catch in this window — what the record shows is guards driven red before their green was trusted, and blind spots, including one guard that went vacuous rather than red. That’s absence of a record, not proof of absence, and it’s the rung whose found-by column went dark. And the mutation rung’s counts remain prose: 94 planted on one story with 21 meaningful survivors, 74 on another. The newest register item about it says what I’d say: a mutation tally that names a number and not its mutations cannot be impeached.

One number I’m declining to publish as a verdict, though the division is trivial: review works out at about $7 per finding and the QA agent at about $9. Eighty-four of those review findings were dismissed as noise and seventy-two deferred; a QA finding is a behavioral defect reproduced on a running build. The denominators aren’t the same kind of thing, and a ratio built on them would rank the instruments backwards.

What a Story Costs, by Phase

The other table that couldn’t exist before the boundary: per-phase cost for every story that merged with labels on. All figures at API list rates (the cash caveat applies: the work ran on a subscription, roughly 10× cheaper). These are the ten stories that had merged with labels on by September 4; post 9 carries all nineteen of Epic 6 and the epic’s totals. Two planning figures here are higher than in this post’s first draft, because the cost query turned out to keep only one model of a session that switched models — post 10 has that story.

What the split shows — first on these ten, then confirmed on all nineteen of the epic:

• Planning is not small. On the combat-domain story, planning cost more than development: a codec story is decided at creation. On the UI stories the ratio inverts. Nothing about the old single-lane cost could have told me which.
• Review is the widest-variance phase, $83 to $507 on these ten and $59 to $733 across the epic, where it out-cost development on seven stories of nineteen. Post 7 makes the case that it is also the least instrumented.
• The QA agent is 3% to 12% of a story and 5.6% of the whole epic, and it ran on fourteen of nineteen. The ones it skipped were rulings or stories with no surface to play — a codec, a capture, a data table — plus one where no QA session ran and the record doesn’t say why. That’s the design: the QA lane is selective, not sequential. It hunts for surprises before a human’s device walk, which is expensive and consequential; the E2E lane protects scripted regressions. Running the agent on every story would spend the budget where the walk already looks.

And the cumulative view since the labels arrived, at the September 21 reading: development $6,165, review $5,233, planning $2,675, QA $862, epic prep $145, retrospective $82. Two of those phases were invisible before — prep work used to vanish into the planning bucket, which the retrospective that added the label described as a measurement scheme’s own categories are a place things hide.

What Each Rung Cannot See

The useful way to read a testing stack is by its blind spots, because the blind spots decide where the next dollar goes.

L0 cannot see the browser. A plate rendered in the wrong half of the screen survived 3,583 fast tests, 24 browser-rung tests, a clean typecheck and a passing E2E run; a screenshot found it. That’s why the browser rung tripled during the HUD stories, and it’s why the device walk exists.

Review cannot see execution. Post 4 put it as reading versus driving, and the score hasn’t moved: every vacuous guard in Epic 6 — three on one story, four on another, nine on a third, two on the most recent — was found by a mutation, none by a reader. Review is the highest-yielding rung by count and structurally blind to the defect family that matters most in agent-written tests.

The QA agent cannot see the phone. It plays the built bundle in a desktop browser at a phone viewport, which finds copy, flow and state defects — and it found a way to settle “is this really the realm’s ground” from the desktop that the human walk no longer has to. But the ground under a character on an actual phone, persistent storage prompts, install UI: those remain the walk’s, and the walk’s findings remain uncounted.

The oracle cannot see what the desktop client never sends. The deviation register’s twelfth entry records that the reference client refuses out-of-range and on-cooldown casts locally, so two whole result codes appear in no capture and can never be diffed. Twelve approved deviations, zero pending — down from thirteen, the register’s first net decrement, after the first entry ever filed with an expiry condition was retired when the blindness it excused was fixed. Post 3’s second reading tells that entry’s two-week life.

The counter cannot see a heading it wasn’t told about — or a bold tag, or a differently worded close-out line. Above, three times.

The pattern: each rung was added when a measured failure showed the rung below it couldn’t have caught the defect. None was added because a pyramid diagram said it should exist. The E2E lane is the cautionary case in the other direction — six minutes per run, local-only, its four findings on one early story the only ones it was ever credited with before the attribution column went dark. The next delta was supposed to say whether it earns its keep. What it said instead is that a fifth of the lane doesn’t execute, for a reason the factory register now names from its own side: the lane can’t put a scene in the world or reach a named quest giver, so specs that need one skip, quietly, forever. A test that skips is greener than a test that fails, and far less useful.

Regenerate This

The phase table is one query against the project’s metrics store, pinned to the snapshot’s evaluation time. max_over_time is load-bearing: a resumed session keeps its id and restarts its counter at zero, so last_over_time read one story’s $52 QA walk as $4.78 before the idiom was fixed.

sum by (story, phase) (
  max by (story, phase, session_id, model) (
    max_over_time(claude_code_cost_usage{project="pwa-wotlk", phase!=""}[120d])))

The model in the inner grouping is load-bearing too, and it was missing from this post’s first draft: a session’s series splits by model, so without it a session that switched models is billed for one of them.

Run counts per layer are count_over_time over the test-run event series, summed by layer and trigger. Run durations are not a query at all, and this is a correction: the first draft quoted medians from quantile_over_time, and they were wrong in an instructive way. Every test run is its own series with exactly one sample, so a quantile over it returns that one run’s duration — each “median” I had written down was one arbitrary run. The durations in this post come from exporting the raw samples and taking percentiles in the client. The tell, for anyone checking their own dashboards: mean and median were identical for every layer.

Suite sizes read the per-kind case gauge, grouped by timestamp so that one coherent run is read. The September 4 per-instrument table is the attribution script’s Found by column, with its known defects; the QA counts are a grep for found-by: qa-agent across the QA reports; the review-count regression is one line of Python running both patterns over the epic’s story files (425 against 636); the E2E breakdown is read from the last local run’s JSON report; and the walk count is hand-derived from prose and labelled as such. All recipes ship in the product repo’s query file when it goes public, alongside the dated delta that this post is drawn from.

Second Reading: September 23 — How the Spine Was Born

The sections above were written from the September 4 delta and refreshed to September 21. This one was added from a fact brief the product repo’s Epic 7 preparation session wrote on September 23, and it fills the gap this post skipped: where the browser rung and the end-to-end lane came from.

The plate in the wrong half of the screen was not the first layout defect to survive a green suite; it was the second. A month earlier, on the quest accept-and-turn-in story, two layout bugs went through about 4,050 green tests — 4,043 at the story’s gates, 4,089 two days later — one after the other. Moving the quest dialog out of the instrument layer gave it a full-screen anchor that stayed mounted with no conversation open, and took the default pointer behaviour, so every tap, pinch and joystick touch on the radar was dead for the whole session: an invisible tap-eater. The fix, the same night, put that anchor first in source order, under the chat strip — and the next morning’s device walk found hand in and close dead beneath it. The commit that fixed the second bug describes it as “my own fix from the night before.” That walk found six defects no test could see, and only two of them were layout. (The two incidents’ counts, 4,050 and 3,583, are not comparable: the later, smaller figure almost certainly counts a narrower set. The story file doesn’t say which, so I’m leaving them uncompared rather than reconciling them by guess.)

Before that day, both fixes could only be guarded by reading the source as text — a test asserting markup order, another asserting a string in a stylesheet. A guard that reads text cannot see what a finger hits. So the browser rung was born, on August 22, in a pull request titled the testing spine: its first test asks the real browser elementFromPoint what a finger would hit, then adds an actual click, and it was proven red on both pre-fix commits in scratch worktrees before it was believed green on main. Even that first test was vacuous three ways before it could go red: without the design tokens the dialog measured 56 pixels and never reached the strip; a minimal fixture was a 124-pixel box, so the test had to use walk-shaped content; and hand in is disabled until a reward is picked. Post 4’s discipline, applied to the instrument on the day it was built.

The same pull request landed the end-to-end lane, and its first spec replaced a step of my walk with a script: find the quest giver’s ”!” by its ink on the radar and tap it; wait for three reward names to arrive (the story’s headline bug, now a retrying assertion); accept, and the ”!” clears; give the quest back, and the ”!” returns. Green three runs in a row at under ten seconds each, after a two-minute build. What my own walk of the same step costs is not measured in seconds; that story needed two of them.

It took five red runs to reach that first green, and every one was settled by evidence from the realm rather than a guess: the realm answers a greet only within speaking distance; a second ink cluster was the giver’s own ”!” two yards away, which the spec had excluded; two givers in range put one centroid between them; movement bursts overshot by eleven yards, so the rates were measured — one second is about eight yards, 300 milliseconds about 87 degrees; and the “before” snapshot had been taken under the dialog’s own scrim. The same day, a single query to the realm read the fact the lane had paid five red runs to learn: the test character parks 2.8 yards from the giver. The rule that came out of it is now in the test lane’s README: ask the realm before you assume it.

Two running totals since September 21, each with its snapshot. L0 stood at 6,894 on the main branch on September 23 and at 7,008 at the tip of the preparation branches, with telemetry readings up to 7,025 that no committed file holds yet. The end-to-end lane ran 48 passed and 34 skipped on two consecutive full runs, with zero parked tests — still 41 tests in two orientations — and a series boundary now sits under that skip count: the lane restores its fixture characters before every run, so skips caused by drift between runs disappear. Two baselines for the last run before the boundary are on record, 46/36 and 44/38, so a comparison has to say which one it’s against. And the per-phase cost changed meaning on September 22 — a single build lane replaced the plan lane — so no post-21 phase figure appears here.

Final Thoughts

If you take one practice from this post, take the label — and take the lesson about which labels last. The day every session said which phase it was in, the factory’s testing stack stopped being a diagram and became a budget: one where I can see that review costs nearly as much as development and finds the most, and that the QA agent costs an eighteenth of the epic and found something every single time it ran. That label is stamped by a script, and it held. The other label — which instrument caught each finding — was a convention people and agents had to remember, and it lasted two stories. Stamp what you can; distrust what you have to remember.

Use test-layer metrics to decide the next verification investment. Never use them to celebrate activity. Six thousand seven hundred green tests and a gesture that cannot work on a phone is not a contradiction; it’s a description of what green means. If I were spending the next dollar, it wouldn’t go to the base of this pyramid. It would go to the middle, and to giving the tip a world it can actually set up.

The series keeps running with the factory. The hub carries the freshest totals, the register from post 7 carries what breaks next, and when the counter is fixed — it wasn’t, between this post’s first draft and its second — this post gets a dated second reading, with the numbers it should have had. Next: what all of this costs, phase by phase, for the first epic the factory could measure end to end. That’s Deciding and Checking Cost More Than Building, in two weeks.