Greens That Prove Nothing

In the last week of this snapshot, a review agent hardened a WebSocket client against a race condition, wrote a test to prove the fix, and watched it pass. Then — because this factory has a rule — it removed the fix and ran the test again. Still green. It rewrote the test more carefully. Green with the fix removed, twice. The only honest outcome was to delete the test and leave a comment saying exactly why, so the next reader finds the story instead of a gap.

A failing test tells you something true. A passing test tells you almost nothing until you’ve seen it fail — and the largest defect family in this project’s findings ledger, a dozen entries and counting, is exactly that: greens that prove nothing. Last installment was about gates with a real oracle on the other side. This one is about the gate’s own material — guards and tests that were green for weeks while asserting nothing at all — and the uncomfortable discovery that reading code, by agents or by me, has caught essentially none of them.

(Series rules apply: this is a mid-product snapshot, and every count below is regenerable from the project’s own records.)

Five Ways a Test Lies

1. The empty input. A design-token guard asserted that the app’s stylesheet contains none of the forbidden legacy values — written as ?raw import plus a set of “must not contain” assertions. But the test runner strips CSS imports, so the stylesheet arrived as an empty string, and an empty string passes every “does not contain” check ever written. Vacuously green from birth; only the positive assertions in the same file failed and gave the game away. The fix went two levels deep: any negative guard must first prove its input is non-empty — and the rewritten test parses the design document’s own source instead of a hand-copied table, because a transcription checked against a second transcription only proves the two transcriptions agree.

2. The self-satisfying matcher. A guard proving that sign-out discards the player’s identity sliced the source starting at function signOut and then asserted the slice contains signOut(). The anchor is inside the haystack: you could delete the entire feature and the guard stayed green, matching its own search term. A sibling asserted an ordering property with indexOf(needle) < indexOf(other) — where the slice began at the needle, so the first index was always zero and any order passed. This shape has now appeared seven times in four stories, including once inside the very review that had just written up the previous instance. Every one of them read correctly.

3. The audit that can’t see its target. A security audit enforced “the gateway reads no secret-shaped environment variable” with three regexes anchored on the literal text process.env. The gateway, deliberately and for good reasons, takes the environment as an injected parameter — so it never writes process.env at all, and the audit had been structurally unable to fire on the unit it was written for since the day it was written. It was caught only when a story mutated the code to the forbidden name and re-ran the audit: it stayed green. A green security check proves nothing until you have watched it go red.

4. The data that can’t discriminate. The oracle fixture used for world-entry comparisons carried a single game character. Against one character, “render in the oracle’s enumeration order” passes for a client that sorts, reverses, or ignores order entirely; “log in with one of the enumerated character IDs” passes for a client that ignores the player’s selection, because there is only one ID it could send. Eight compared cases, green since two stories earlier, two ratified decisions unprovable the whole time. The sister case is crueller: an acceptance criterion prescribed verifying a logout packet “byte-equal to the captured frame” — for an opcode whose payload is zero bytes. Comparing empty to empty passes on any implementation, including one that never sends the packet — and that AC explicitly cited the one-character finding as the reason it existed. Presence and adequacy are different questions about a fixture: when a decision is about ordering, multiplicity, or selection, the oracle needs at least two of the thing, and that’s a capture requirement, not a test-writing one.

5. The subject that was never there. Two rotation tests proved that an error screen outranks the game surfaces by asserting those surfaces were absent after rotating. They were absent — because the test’s setup helper stopped one click short of the screen that mounts the orientation router, so the rotation event fired at exactly zero listeners and nothing was ever going to be there. A negative assertion is only as strong as the proof that its subject could have been present; the durable fix was asserting the listener count inside the setup helper, so every test in the file inherits proof that its input exists.

It Survived Everyone

The tempting diagnosis is “agents write sloppy tests.” The record doesn’t support it. The zero-byte acceptance criterion survived story creation, an adversarial review round, and my ratification — while citing the previous vacuity finding as its own justification. The ordering guard with the always-zero index was written by an agent that had read the two prior postmortems that same day. The deleted test happened inside the review that had just filed the sixth instance. Ledger entry F-77 states the score plainly: reading has failed to catch seven of these; driving has caught all seven.

The reason is structural, not motivational. Vacuity is a property of execution, and review is an evaluation of text. A line like expect(body).toMatch(/signOut\(\)/) reads as “assert the body calls signOut” — the intent is visible, the mechanism isn’t, and nothing about the line looks wrong. Reviewers (human or agent) judge whether a test is plausible. Only running it against a broken implementation judges whether it discriminates. Those are different questions, and the entire family lives in the gap between them.

Which means awareness doesn’t fix this. Writing “this is the F-40 shape, don’t do it” into a document does not immunize the document — one of these traps sat inside the warning about the trap, and the warning arguably made it worse, because it read as evidence the author had checked.

Drive It Red

What works is a team agreement, adopted in epic 1 and re-earned since: assert red stays red before asserting green is green. Every new guard is shown failing — by breaking the thing it protects, not by breaking the test — before its green is believed. The refinements that turned a slogan into a discipline:

• Order matters. When a story adds a guard and removes the violation it guards against, drive the guard red first, against the tree that still contains the offence. Written afterwards, it is green from birth, and a guard that has never failed is indistinguishable from a guard that cannot fail.
• Pick discriminating mutations. An early red-path test flipped a byte inside a float that carried a comparison tolerance — the mutation landed inside the tolerance and honestly passed. Mutate a field the gate has no permission to forgive.
• Pair every ban with its positive half, sharing one pattern. “Nobody may read orientation except the router” is satisfied by a world where nobody reads orientation, router included — the feature is gone and the suite is green. The paired assertion “the router actually reads orientation,” using the same regex, is what caught the pattern hole that matchMedia?.( had slipped through. One pattern, exercised in both directions, fails on the side where the answer is known.
• Make helpers prove their fixtures. Setup code asserts the shape it claims to produce — the listener is mounted, the source slice is non-empty, the generated negative input actually has the property the test needs. Emptiness becomes a failure instead of a free pass.
• Delete what can’t fail. A test that survives the removal of what it tests is not weak evidence, it’s false evidence — it records “verified” for something that is merely present. Deletion plus a comment is the honest ledger entry.

And one caveat the discipline taught about itself: driving a guard red proves it can fail, not that it covers every form it claims to. The orientation guard was provably failing and still had a hole in its regex. Red-first is the floor, not the ceiling — which is why the positive-pair rule exists.

Regenerate This

No telemetry queries this time — this post’s evidence is documentary. The twelve findings are entries F-13 through F-77 in the project’s findings ledger; each driven-red mutation is recorded in the corresponding story file’s Dev Agent Record, with the witnessing test output quoted. All of it ships with the product repo when it goes public, and the counts above will be stale by then — that’s the running-total rule working as intended.

Second Reading: September 4

Eight more stories have merged since the count above was taken, and the family kept growing — by the stories’ own tallies, three vacuous guards on one story, four on the next, nine on a third, and two on the most recent, story 6.15. Those last two are worth the retelling, because each is a shape the five above don’t cover.

The test that never ran its subject. A guard for a QA helper asserted that a forbidden path made the script throw — expect(...).toThrow(). The dev session deleted the guard the test protects, and the test stayed green. Two causes stacked: toThrow() can’t tell a refusal from any other death, and the module that runs the script was never imported in that file, so the test had been passing on a ReferenceError since the day it was written. It had never executed the script at all. The rewrite reads the child process’s stderr and pins the refusal sentence, and was driven red twice — guard deleted, then guard rewritten as the naive check the traversal case exists to catch.

The observer that coalesces the defect away. A regression test for a button that briefly rendered idle between two operations watched the DOM with a MutationObserver — and was vacuous, because an observer callback is itself a microtask, so a state set and reverted within one synchronous run is merged into nothing before the observer fires. The planted defect was invisible by construction. The replacement samples the label once per task turn, which is the only kind of frame a browser could ever paint.

Both were caught by a single habit the discipline above didn’t yet name: verify the mutation landed before believing the pass. A mutation that didn’t mutate produces the same green as a guard that can’t fail, and nothing about the run distinguishes them. That check is now the first thing the factory’s new mutation tool does — and the tool exists because the manual version had a second cost: a session undoing a mutation with git checkout -- destroyed an unrelated uncommitted change beside it. Snapshot, assert landed, run, report, always restore; a green suite on a landed mutation is the failure. Post 7 tells how it got built.

And the number that reframes the scoreboard: on story 6.15, a fourth review layer that asks only “what mutation turns this assertion red?” found more real defects than the three standard review layers combined — on a diff where 3,200 of 7,734 added lines were tests. Reading: still zero. Driving: still everything.

Final Thoughts

If agents write your tests, the tests are part of the surface you have to verify — not the verification. That sounds expensive; it isn’t. The whole discipline is one deliberate red run per guard, minutes each, and in this project it has a perfect record against a failure mode that defeated every reading-based defense, mine included.

The deeper lesson is about where trust can come from at all. A green suite is a claim. The cheapest falsification of a claim is to break the thing it describes, once, and watch the claim notice. Everything in this post was ultimately catchable inside the repository by someone willing to do that. The next post is about the defects that weren’t — the ones no repository test could see: a CDN quietly caching the service worker that exists to fetch updates, a manifest request that arrives without credentials at an auth wall, a browser that retired the install prompt an acceptance criterion depended on. That’s What Only Broke Live, in two weeks.