The Oracle Is the Spec

The uncomfortable truth about agentic development is not that agents write bad code. It’s that the apparatus we normally use to decide code is good — tests, reviews, CI — was designed on the assumption that writing code is expensive and someone honest is watching. In a factory where the workers are tireless, confident, and occasionally generous when grading their own homework, verification is the product. This post is the tour of that product: the behavioral oracle, the three gate layers, and the piece I’d most want to transplant into any organization betting on agentic delivery — the deviation register, the artifact that turns “close enough” into something signed and machine-checkable.

(Series ground rules apply: numbers here are a mid-product snapshot, regenerable from telemetry, and metered at API list rates — actual cash was a ~$100/month subscription plus ~$100 in credits, roughly 10× less than the meter reads.)

The Problem: Accepting Work You Didn’t Do

Every conversation about trusting agent-written code eventually arrives at “but it has tests.” It does — thousands, in this project’s case. The problem is who wrote the tests. When the same agent writes the feature and the tests that pass it, a green suite is a statement of internal consistency, not correctness: the agent proved that its code does what its code does.

The way out is to define correctness somewhere the agent can’t reach. Azeroth Pocket has an unusual advantage here: the product is a mobile web client for a 2008 game server, so a reference implementation of “correct” already exists — the original desktop game client, speaking its wire protocol against the same server. That client doesn’t care what my agents believe. It is an oracle: an external system whose observable behavior is the specification, packet by packet. The factory’s core rule follows: for any protocol-visible behavior, the agent’s output is recorded and diffed against a real capture of the desktop client doing the same thing — and protocol fixtures are captured, never fabricated.

You probably don’t have a 2008 game client to diff against. But you almost certainly have something your agents’ output must match: the legacy system being strangled, the API your partners integrate against, the regulatory calculation with worked examples. The transferable idea is that the strongest specification is a running system the agent didn’t write, and it’s worth real engineering to put one at the center of your gates.

Three Days, Six Stories, No Product

Epic 1 of the project built no product at all. Six stories, July 16–18, built the machine that would verify the product: capture tooling, diff engine, replay runner, deviation register, CI merge gate.

Spending the project’s first three days on verification before product sounds like discipline; it’s actually economics. Post 1 showed a story costs about $92 of metered compute to build — so re-building one wrong is cheap, but not noticing it’s wrong compounds into every story stacked on top. When production is cheap, inspection is where the leverage lives.

The size data makes the same point from the other direction. The median story in this project lands a +5,397-line diff, and the largest ones carry tens of thousands of lines — dominated, to be fair, by committed oracle captures: generated evidence, not authored code. I’ll say plainly what those numbers imply: nobody line-reviews that, and I didn’t. The assurance model was never “the human reads everything” — it’s the machine gates and adversarial layers this post describes, plus device walks, with my attention spent on the staged decisions those layers escalate. A factory producing at this rate needs a quality department, not a proofreader.

The Gate Stack: L0, L1, L2

The machinery arranges into three layers, and the differences between them are about trust, not thoroughness:

Layer	What runs	Where	What it proves
L0	Unit tests, protocol-fixture tests, import-graph rules	Every commit, hermetic	The code agrees with itself and the architecture
L1	The replay harness against a live test realm	The dev loop, on demand	The behavior matches the oracle, per the agent’s own run
L2	The same harness, re-run by CI	Every pull request	The behavior matches the oracle, per someone who isn’t the agent

L0 is conventional, with one twist worth stealing: the architecture is executable. The dependency rules — apps never import apps, the domain layer imports no framework code — are enforced by ArchUnit-style checks seeded in the repo’s first commit, so an agent that “simplifies” by importing across a boundary gets a named rule violation, not a merged compromise. (One measured side effect worth noting: across the code seats, test lines run 1:1 with non-test lines — a ratio the factory has maintained without ever being given a coverage target.)

L1 is the interesting layer. The harness loads a recorded oracle scenario, connects to the live system exactly like a player would — same wire contract, same browser-standard WebSocket API, no test side-doors — replays the scenario headlessly, and diffs what came back against the recording. One command, no manual steps, results pushed to the metrics store under the story’s issue number.

L2 is deliberately unimpressive: it is L1 again, run by CI on every pull request, with the harness’s exit code as the merge verdict. No new checks — the same comparison, re-run by an executor the agent doesn’t control. That redundancy is the point. The local green is the agent proving the work to itself; the CI green is the work proven to the factory. Between them sits the sentence the whole design hangs on: an agent cannot grade its own homework.

One piece of discipline binds the stack together: every story declares its gate class before implementation — Behavior-mirroring (owes a green harness run), Oracle-blind (UI and tooling the oracle can’t see; owes a recorded human review), or both. A merge check verifies the declaration exists, so no story reaches done without anyone having decided, up front and in writing, what “verified” was going to mean for it.

Truth, Captured; “Equal”, Governed

The oracle fixtures are recordings of the real desktop client driven by a human against the reference server — never bytes an agent transcribed from documentation. Every capture-shaped task carries a standing instruction that generalizes to any agentic pipeline: “I couldn’t do this” is an acceptable answer; fabricating the artifact is not. An agent told in advance that honesty is the cheap path doesn’t have to choose between failing and inventing — across the epic, five tasks only a human could perform were all either done for real or declared undone, none quietly simulated.

Comparing two honest recordings raises the harder question: they’re never byte-identical. Cryptographic handshakes differ every session by construction; database-assigned IDs differ; positions drift by float-rounding amounts. The entire meaning of “oracle-equivalent” therefore lives in what you choose not to compare — exactly where a gamed gate would hide. So those choices are an artifact, not code: the Comparison Spec, a versioned document naming compared messages, masked fields, and tolerances, each with a written reason — because an over-broad mask is a gamed gate wearing a lab coat. Changing it is reviewed like a deviation; an agent doesn’t get to widen what “pass” means in the same pull request that needs the pass.

And the diff engine carries one rule I’d port to any verification system tonight: comparing nothing is an error, not a pass. An empty case list — a typo’d spec, a wrong fixture path — exits non-zero and writes no result. An oracle that tested nothing proved nothing, and the tooling refuses to let silence look like success.

The Deviation Register: “Close Enough,” Signed

Some differences from the oracle aren’t defects — they’re the product. A browser cannot open a raw TCP socket, so a gateway performs the cryptographic handshake on the client’s behalf: a permanent, deliberate difference from how the desktop client behaves. Every real system that mirrors another one has a pile of these, and in most shops that pile lives in tribal knowledge, code comments, or a wiki page nobody re-reads. The question a verification model has to answer is: where do approved differences live, and what stops that category from becoming the place failures go to be forgiven?

The answer here is the deviation register: a human-readable table that is simultaneously a machine-readable artifact the harness enforces, fail-closed in both directions:

• No entry → the behavior must match the oracle. A detected difference matched by no rule fails the gate, full stop.
• Entry without an approver → authorizes nothing. A difference matched by an unapproved entry still fails, and the failure names the entry — the fix path is get it approved or match the oracle, never “it’s documented somewhere.”

Each entry carries matching rules in a deliberately impoverished vocabulary: exact case IDs, exact field paths, two rule kinds, and that’s all. No wildcards, no whole-connection matches, no “everything else” paths. That poverty is a security decision: an expressive matcher would let one generous entry silently swallow a whole connection’s worth of future regressions — the gamed gate arriving through the file format instead of through a bad approval. A counter-metric watches the register’s size, on the theory that a register that grows every time the gate is inconvenient is a gate being negotiated with.

The register earned its sharpest rule on July 19, the first time the harness compared a real agent-built gateway against the oracle. Six cases: the four cryptographic ones passed on merit — the first non-tautological green in the project’s history — but two were waived by a matching rule written before the behavior it described existed, a plausible guess that happened to match a difference with an entirely unrelated cause. Green, 6/6, a third of it unearned. The repair is the instructive part: the guessed rules were removed, not refined, and the format was tightened so an entry may carry zero matching rules — still documented, still approved, but authorizing nothing until someone has observed the real behavior. The standing principle: a rule should describe behavior someone has actually observed. Several entries today deliberately match nothing, and that’s a feature — “this deviation has no observable signature in the comparison we run” is a stronger statement than any invented signature.

Concretely, this is the entire machine half of the register at the August 8 refresh — six entries, all approved by name, zero pending, unchanged through the last seven stories:

{
  "matchVersion": 2,
  "entries": [
    { "id": "DEV-001", "match": [] },
    { "id": "DEV-002", "match": [] },
    { "id": "DEV-003", "match": [
      { "kind": "field-difference", "caseId": "auth:REALM_LIST:s2c",
        "fields": ["realms[].address"] }
    ]},
    { "id": "DEV-004", "match": [
      { "kind": "field-difference", "caseId": "auth:AUTH_LOGON_CHALLENGE:c2s",
        "fields": ["accountName"] },
      { "kind": "field-difference", "caseId": "world:CMSG_AUTH_SESSION:c2s",
        "fields": ["username"] }
    ]},
    { "id": "DEV-005", "match": [] },
    { "id": "DEV-006", "match": [] }
  ]
}

That’s it. Six approved architectural departures from the reference — a gateway doing the cryptography, a realm registry in browser storage, a dual identity model, auto-reconnect, a keepalive substitution — and the sum of what they’re allowed to forgive is three exact field paths on three exact cases. Everything else must match the oracle or fail. Each entry’s human row carries the full rationale; each rationale note records why its rules have the shape they have, including the removals.

The register’s best property only shows over time, so follow one entry’s life. DEV-006, the keepalive, was born from evidence: the desktop client survives idling on its ambient traffic, a clientless pipe has none, so the entry authorizes a substitute heartbeat — with the rationale verified in the game server’s source, down to the disconnect hazard that copying the oracle’s own ping would have imported. Weeks later, a story made the harness emit real movement for the first time, and the obvious inference was that the heartbeat had become redundant — so the entry was re-read against the new evidence, and the conclusion (it stands: two movement frames at the start of a session are not ambient traffic, and the live run proved it) was recorded in the entry itself. Best of all, the entry now carries its own falsification criterion: it names the future story shapes that would genuinely weaken it and the cheap discriminating test to run when they arrive. The register, not tribal memory, schedules its own re-examination.

What you end up with is something rare in software and routine in regulated industries: a complete, current, machine-enforced ledger of every known divergence between the system you built and the system it answers to — who approved each one, why, and exactly what it does and doesn’t excuse. If your organization answers to auditors, that sentence is the whole pitch; post 6 returns to it.

Where It Still Takes a Human

One honest asterisk. The L2 gate runs on every pull request and shows its verdict — but the enforcement half hit a mundane wall: GitHub grants free personal accounts no branch protection on private repositories, so the story whose thesis is “an agent cannot grade its own homework” ended with the human guarding the merge button by hand (F-13 in the project’s findings ledger). The decision was to accept it, write the price down, and set a revisit trigger — a bar moved in writing, with its cost attached, is still governance; a bar moved silently is a gate rotting. A related review finding (F-14) supplied the durable lesson: markers asserting “a human did something” must be structural claims — a real URL at an anchored position — never string matches an agent could satisfy by pasting the template.

Regenerate This

The per-story costs behind the $207.61 come from the reset-aware per-issue aggregation described in post 1. The oracle-run history — every green and red, per attempt, per trigger — is an event series:

last_over_time(oracle_run{project="pwa-wotlk"}[90d])
last_over_time(oracle_case_pass_ratio{project="pwa-wotlk"}[90d])

The register’s size over time is deviation_register_size. The Comparison Spec, the register with its matching rules, and every story file quoted here are files in the product repo, quotable verbatim when it goes public.

Final Thoughts

If you’re building or buying an agentic pipeline, the four properties I’d insist on are all cheap in mechanism and expensive only in discipline. Define green outside the agent — an oracle system, a reference implementation, a contract the agent can’t edit — and treat every softening of “equal” as a versioned, human-approved artifact. Fail closed everywhere: comparing nothing is an error, an unparseable register authorizes nothing, an unapproved entry authorizes nothing. Never trust a gate you haven’t seen fail. And make approvals structural, not strings.

Everything in this post concerns the gate the project got right — the one with an external oracle on the other side. The next post is about the tests that had no oracle: guards that stayed green for weeks while asserting nothing at all, an audit that couldn’t fire on the code it was written for, a security check an empty string satisfied. The count found so far is in double digits, and reading the code caught almost none of them. That’s Greens That Prove Nothing, in two weeks.