Planning a Product with an Agent Factory

Planning the product cost $177.80. That bought a product brief, a PRD with 32 functional requirements, a two-document UX specification, an architecture spine with 18 logged decisions, six epics broken into 41 stories, and a readiness report certifying the whole stack aligned — produced in six days by the BMad-METHOD planning pipeline with me in the approver’s chair.

The interesting question isn’t how it got that cheap. It’s whether the output was any good — and the honest answer comes from the part of the pipeline I almost skipped: the adversarial review passes. They cost minutes each. They found the defects I had already read past. This post is the tour of what a $177.80 planning corpus contains, where the machines failed, and why the cynical second pass is the ceremony that earns its keep.

(As with everything in this series: running totals, not final ones — planning didn’t end when implementation started, and the numbers below cover the initial corpus, July 10–16.)

Why BMad at All?

Let me be upfront about something, because the price tag makes the methodology look like an easy recommendation: as a solo developer, I don’t recommend BMad. It is a lot of ritual for one person — role-played phases, hand-offs between personas that are all, ultimately, me approving my own paperwork. If this project were only about shipping Azeroth Pocket, I’d have planned it in a fraction of the ceremony.

I ran it anyway because the experiment was never really about a solo project: it’s a rehearsal for transporting the method into a company. And in that setting, BMad has two properties that most “AI coding” setups lack. First, it’s structured around a company’s actual roles — analyst, product manager, UX designer, architect, scrum master — so a real organization can slot real people into the loop at the point where their judgment matters, rather than funneling everything through whoever happens to be comfortable with a terminal. Second, and more importantly: code is not the deliverable. Every frontier model can write code now; that part is table stakes. What an organization actually runs on is everything around the code — the requirements document, the architecture decisions, the tickets, the traceability from one to the next. BMad is spec-driven development taken seriously: the corpus below is the product of the planning line, and the code that follows in later posts is downstream of it. That’s the property worth the ritual — and the ritual’s cost, at company scale, is spread across the roles it was designed for instead of concentrated on one exhausted approver.

Six Days on the Assembly Line

BMad structures planning as a sequence of role-played phases, each an agent workflow with its own instructions, each consuming the previous phase’s artifacts. Mine ran: product brief → PRD → UX → architecture → spec → epics & stories → implementation-readiness check → sprint planning. The metered cost, day by day, from the project’s telemetry:

UTC day	Cost	What happened (anchored to git/GitHub, not memory)
Jul 10	$22.43	Epic-1 issue opened. The brief itself ran in Gemini and Claude Desktop — unmetered
Jul 11	$12.77	prd.md committed 02:04
Jul 12	$20.73	”ux bmad done” committed 03:08
Jul 13	$0	no samples — a day off
Jul 14	$23.46	architecture in progress
Jul 15	$59.46	”bmad architecture” committed; the ~10-minute spec phase hides somewhere in here
Jul 16	$38.95	epics + 41 stories + readiness report committed 01:28
Total	$177.80	72.0M tokens

Two honesty notes on that table. The phase boundaries are honest to day resolution only — the PRD, UX, and epics all landed on small-hours commits, so their work straddles days, and July 16 is mixed with the first story-creation sessions. And the brief phase is a measurement blind spot: it ran through Gemini’s UI (as a deliberate test of a BA-friendly, non-terminal interface — the agent persona “Mary” interviewed me, and the output came co-signed “Author: David & Mary (Strategic BA)”, which raises a genuinely open governance question about who signs AI-drafted artifacts) and was then rewritten in Claude Desktop. Neither lane is metered. Not measured — the $177.80 starts where the telemetry does.

The Pass That Earns the Ceremony

Every phase in this pipeline ends with an optional adversarial review: a fresh agent, prompted as a cynical senior engineer, told explicitly which decisions are already adjudicated and not up for relitigation — and set loose on the artifact. I ran one on nearly everything. They are the strongest earned value in the whole planning corpus, and I can show it, because their outputs are documents I can quote.

The PRD pass found 25 findings — 4 critical, 10 major, 11 minor — in a document a careful human and a first review pass had already accepted. Its verdict, verbatim: “Not ready to finalize.” The four criticals are worth naming, because none of them is a nitpick:

1. A requirement that violated the game’s own physics. The PRD promised push notifications for messages received “while the app is closed” — but the 2008 game protocol has no offline message delivery; a whisper to a logged-out character simply fails. The promise was physically unimplementable without a major unnamed architectural component. I had read that requirement at least twice without seeing it.
2. The PWA-on-LAN trap. Service workers, install prompts, and badging all require a secure context; the deployment story said “open the LAN URL in Chrome.” Half the feature list silently depended on an HTTPS story the PRD didn’t have. (This finding is why the product ended up behind a Cloudflare tunnel — the fix became architecture.)
3. The central term of the whole verification model was undefined. “Oracle-equivalent at packet level,” taken literally, is unachievable — session keys differ per session, timestamps and GUIDs vary, the desktop client emits packets a web client never will. Without a specification of which opcodes are compared and which fields are masked, every merge gate in the PRD was either unfalsifiable or guaranteed to fail. The review called this “the actual hard product decision” — and it was right: that demand became the Comparison Spec and the machine-readable deviation register, the subject of the next post.
4. Players die, and the PRD didn’t know it. The daily-loop success metric required combat; combat guarantees death; the death flow — corpse run, resurrection, the first stuck family member — had no requirement, no UI surface, nothing. The most certain event in the product’s life was unplanned.

The UX pass did the arithmetic the design skipped. The experience spec promised a combat arc of “a flat 8 abilities, all visible, 56px keys, one-handed at a 360dp viewport.” The cynic reviewer computed it: eight 56px keys need 392px of arc path; the arc that stays clear of the radar offers about 104–136px — room for three. Even ignoring the radar entirely yields at most 7.8 keys, and the design’s own mockup, it noted, only ever drew six keys, overlapping, on a canvas that overflowed the stated viewport. A geometric impossibility, shipped in a spec both the design agent and I had signed off on, caught by a reviewer that cost about ten minutes and 105k tokens. The product has a six-ability arc today because of that pass.

The architecture pass got a different lens: verify every claim against the live world, not training data. Its verdict deserves quoting because it names a failure mode every LLM-planning pipeline has: “the Stack table reads like a late-2025 training-data snapshot” — three rows named superseded versions for a greenfield mid-2026 project, and one row mischaracterized a library as “current (official)” when it had been dormant since 2021 and couldn’t do what the architecture needed it for. Nothing invalidated a decision; everything needed a currency pass. An agent’s confident version number is a hypothesis, not a fact.

The epics pass produced 18 challenges; I ruled on each: 13 accepted, 3 accepted as wording bugs, 1 partially rejected, 1 rejected. Sixteen acceptance-criteria-level fixes, no structural changes — which is itself informative: by the time the pipeline reached epics, the upstream documents were strong enough that the cynic found copy-editing, not architecture.

The pattern across all four: the adversarial pass catches what the author and the approver miss, because both of them read the document believing it. The cynic is prompted not to. That stance turns out to be worth more than the model’s intelligence — it was the same model in every seat.

Claude, Meet Claude

One experiment went further: I took the finished architecture and handed it to a different Claude instance — one whose context was my homelab, the infrastructure this product must actually run on — and told it to challenge the architect. Five inconsistencies surfaced; the architect acknowledged about 90%, adapted the rest without impacting the homelab side, and the exchange converged fast. One flag — a timeout-flake failure mode in the CI design — neither instance would have caught alone: it lived precisely in the seam between the two contexts.

I went in skeptical of Claude-reviewing-Claude, and my skepticism had a specific shape: in my experience with earlier model generations, two agents given contradictory objectives don’t converge — they trench in, each optimizing its own brief, and the loop produces politeness or stalemate rather than synthesis. This exchange was the first time I watched the opposite: on the point of real tension, the two instances found a resolution that genuinely satisfied both briefs at once, and it was a better answer than either had proposed alone. Genuinely satisfying to watch.

The rule I took away, and now apply generally: same model is fine; same context is the thing to avoid. A reviewer’s value is the information asymmetry it brings, and you can manufacture that asymmetry by giving the same model a different vantage point.

Where the Machines Failed

The planning run also produced a catalog of agent failure modes — each caught, each cheap here, each expensive if your organization plans products this way without knowing them.

Elicitation absorbs whatever expertise you leak. The BA workflow happily accepted my architect vocabulary during the brief interview instead of deflecting it to a later phase. Harmless when the stakeholder is also the architect; corrosive when a real business analyst’s half-remembered technical opinions get soaked into a requirements document as facts.

LLMs overclaim compliance unprompted. “Absolute legal isolation.” “Completely mitigates.” “Eliminates the hardest security problem.” Two different vendors’ models produced this register of absolutism on their own, and I rewrote every instance down to what was actually true (the shipped phrasing is “mitigation, not legal safety”). In a regulated environment, one unnoticed sentence of this kind in a spec is a future audit finding.

Agents overturn adjudicated decisions without flagging the conflict. A one-day Copilot spike, asked to explore a side question, silently replaced an already-decided security approach with a different one and contradicted a settled scope ruling — no mention that either had been decided. The countermeasure became a standing governance rule of the factory: agents may propose reversals, but must mark them as reversals. (You’ll meet this rule again in the governance post; it’s load-bearing.)

A warning written down is not a fix. The readiness report — the pipeline’s final gate, which came back ✅ READY, 100% requirement coverage, zero critical issues — flagged exactly one leftover as having “real defect potential”: a stale instruction in the UX journey telling the app to ask for notification permission, contradicting a later ruling that removed that feature. The flag was accurate. It was also merely a flag: the stale sentence stayed in the PRD, and three weeks later a story built from that document nearly shipped the forbidden permission prompt — an agent following its instructions faithfully into a defect. The eventual fix was mechanized (a contract test that scans the built bundle for notification APIs), and the surface lesson generalizes to every document-driven pipeline: when a ruling supersedes a document, patch every document it touches or enforce the prohibition in code — a note in a third file is not a fix.

But the deeper diagnosis is the one I’d want a methodology designer to hear, and it’s the same principle I argued in Dual-Engine Knowledge System: the goal is a single source of truth for every piece of information, and duplication is the disease. This is BMad’s structural flaw. Its document taxonomy — brief, PRD, UX pair, architecture, epics, distilled context files — doesn’t always let one fact live in exactly one place, so the same requirement gets restated across four documents, and every restatement is a future contradiction waiting for the ruling that updates three of them. The stale-notification defect wasn’t an oversight; it was the architecture of the corpus expressing itself. A spec-driven pipeline is only as trustworthy as its deduplication discipline, and that’s a property to design in, not to patch after.

Regenerate This

The daily planning costs come from the no-issue telemetry lane, windowed by UTC day:

# /api/v1/query_range, step=1d, July 9 – August 3
sum(increase(claude_code_cost_usage{project="pwa-wotlk", issue="0"}[1d]))

— with the usual caveat from post 1 that increase() runs ~1% high and publication numbers come from the reset-aware export. The phase anchors are git commit hashes and issue timestamps, none from memory. The adversarial review documents quoted above are files in the repo, quotable verbatim when it goes public.

Final Thoughts

The number everyone will quote is $177.80. The number I’d actually plan around is $4.34 — planning a story costs about 5% of building it ($92, from post 1), which means the old economics of skimping on planning are gone. When the planning corpus is this cheap, the binding constraint becomes review bandwidth: my time in the approver’s chair, reading, ruling, catching what the machines wrote confidently and wrongly.

And that’s why the adversarial passes are the real finding of this post. They are the only part of the pipeline that scales my side of the work — a ten-minute cynic that reliably catches what I miss on a document I believe in. If you adopt one thing from this factory’s planning line, adopt that: never let the author’s model be the last model that reads the document, and never let the reader share the author’s context.

Next installment: the verification machinery those planning documents demanded — behavioral oracles, three gate layers, and the deviation register that turns “close enough” into a signed artifact. That’s The Oracle Is the Spec, in two weeks.