What It Cost

My dashboard told me story 3.5 cost $60.80. The truth was $138.02.

Nothing was broken. Every sample in the metrics store was accurate. The query was wrong in a way that produced a plausible number instead of an error — and a plausible number is the most dangerous thing a dashboard can show you. Summed across the project, the naive query under-reported total cost by a factor of 2.6.

This is the first installment of The Agent Factory, and it’s the bill: what 20 production stories cost when an agentic pipeline builds them, where every dollar and token went, which model actually did the work — and why the meter itself had to be debugged before any of those numbers could be published. The editorial rule for the whole series applies from the first line: numbers I can’t regenerate are numbers I can’t publish, and where the data doesn’t exist, I’ll say not measured instead of estimating.

The Instruments, and How They Lied

The setup is simple: Claude Code’s OpenTelemetry exporter ships cost and token counters to a VictoriaMetrics instance on my homelab Pi, and a small wrapper stamps every session with a project and issue label — so each GitHub issue, and therefore each story, carries its own meter. A second reporter records verification results (oracle runs, pass ratios) with matching labels. On paper, this answers the question every agentic-development claim should have to answer: what did this story cost, and how many attempts did it burn?

In practice, the instruments failed in three distinct ways, and each failure taught me more than the numbers did.

The counters reset, and nothing says so. The cost and token metrics are per-session cumulative counters — and they reset to zero when a new session starts, on the same series, with no session identifier to tell the sessions apart. A max_over_time query therefore keeps only the largest single session and silently discards the rest. That’s the $60.80: story 3.5’s biggest session, masquerading as its total. The fix is to export the raw samples and re-derive sessions from the counter resets themselves — every monotonic run of the counter is one session; a decrease starts the next one — then sum each session’s final value. VictoriaMetrics’ own increase() function handles the resets and lands within about 1% of the truth (it read $139.43 for that story), which makes it a fine sanity check and a poor publication source.

Two quality gauges read “perfect” for the entire project, and they were working as designed. The reporter emitted oracle_first_pass and oracle_attempts_to_green — designed to be the y-axis of the whole cost-versus-quality analysis. Both read exactly 1, for every story, over their entire history. Not because the factory was flawless: story 3.2 took five red CI attempts before going green, and the per-attempt data says so plainly. The gauges were write-once values keyed on the first recorded run, and the dev loop only records a run once it’s already green locally — a definition with structurally no variance. They measured a tautology of the workflow, correctly, forever.

That finding deserves its one-line form, because it generalizes: a metric that has never been observed to vary has the same evidentiary value as a test that has never been observed to fail — none. The testing discipline this factory lives by (“drive every guard red before believing it” — post 4’s whole subject) has a telemetry form: show the metric moving before trusting what it says. Both gauges were retired the day this was found.

Even the audit produced a false claim, by the same mechanism one level up. While compiling these numbers I declared a metric absent from the store — off a discovery query that, without an explicit time range, silently scopes to a recent window. The metric existed; it was just sparsely written. A query whose time window you did not choose is answering a question you did not ask. Caught before publication, recorded in the query log, and the reason this series inlines its queries.

The Bill

With the reset-aware summation in place, here is what the factory cost, from the first planning session on July 10 to the snapshot on August 2. Be clear about what this is: a mid-product snapshot, not a final bill. Three of six planned epics have shipped; the radar-combat milestone, three more epics, and the eventual 3D client are all still ahead, and this series updates its totals as they land.

Two honesty notes before anyone quotes these numbers.

First, the meter and my bank account disagree by roughly 10×. The exporter prices usage at API list rates. The work actually ran on a Claude Max subscription — about $100 a month, plus $100 in one-off credits when I ran hot. So the $2,479 is what this compute would have cost pay-as-you-go, and it’s the honest number for anyone estimating the approach at API prices; my actual out-of-pocket was on the order of a tenth of it. Both numbers are true. Quoting either one alone would be misleading in opposite directions.

Second, not everything was measurable, and here’s the residue. A $258.62 lane (288M tokens) carries no project or issue label at all and cannot be attributed to any repo from telemetry. The related homelab work that supports the factory metered $367.14 in the same period — real, adjacent, and not counted in the project total. GitHub Copilot handled one planning spike: its gross meter peaked at $5.78 and its net meter reads $0.00, because the plan’s allowance covered it — that zero is the finding, not missing data. And four whole lanes — Gemini (the initial product brief interview), Claude Desktop, Claude’s design tool, and some desktop sessions — have no telemetry at all. Not measured. The blind spots are part of the measurement.

Twenty Stories, Story by Story

Every story, its verification class (Behavior-mirroring stories are checked packet-by-packet against the real game client — the oracle; Oracle-blind stories get mandatory human review instead — post 3 covers the machinery), the model that did the work, and its oracle record:

Story	Gate class	Main model	Cost	Oracle runs (green/red)
1.1	Oracle-blind	Sonnet 5 + Fable 5	$25.18	none recorded
1.2	Oracle-blind	Sonnet 5	$26.47	none recorded
1.3	Oracle-blind	Sonnet 5 + Fable 5	$43.71	1g
1.4	Oracle-blind	Fable 5 + Sonnet 5	$44.32	1g / 1r
1.5	Oracle-blind	Fable 5 + Sonnet 5	$49.71	1g
1.6	Oracle-blind	Opus 4.8	$18.22	4g
2.1	Behavior-mirroring	Opus 4.8	$83.26	2g
2.2	Behavior-mirroring	Opus 4.8	$120.35	4g
2.3	Oracle-blind	Opus 4.8	$60.06	1g
2.4	both	Opus 4.8	$127.00	4g
2.5	Oracle-blind	Opus 4.8	$77.50	2g
2.6	both	Opus 5	$156.11	2g
2.7	both	Opus 5	$111.99	3g
2.8	Oracle-blind	Opus 5	$109.30	3g
3.1	Oracle-blind	Opus 5	$215.13	3g
3.2	both	Opus 5	$163.46	3g / 5r
3.3	Oracle-blind	Opus 5	$61.80	3g
3.4	both	Opus 5	$117.05	9g
3.5	both	Opus 5	$138.02	4g
3.8	both	Opus 5	$82.80	5g

Three things in this table earned their place in the series.

Verification is not the cost driver. The natural assumption is that oracle-checked stories cost more — all that harness machinery, all those comparison runs. The data says otherwise: within the Opus 5 era (where the model variable is held constant), the Oracle-blind stories averaged $128.74 and the oracle-checked ones $128.24. Statistically indistinguishable. What actually drives cost is scope: the priciest story of the whole project (3.1 at $215.13 — the PWA installation story, whose findings fill post 5) was Oracle-blind. You pay for the size of the problem, not for the rigor of the check.

Red attempts are visible in money. Story 3.2 is the only story with a CI failure streak — five reds before green, each at a pass ratio of 0.875 — and at $163.46 it’s the second-priciest of its epic. The recorded quality gauge, remember, reads attempts-to-green = 1 for this story. The per-attempt series carries the truth; the summary gauge laundered it.

What’s deliberately missing: hours. The telemetry can bound each story between its first and last sample — story 2.6 spans 40.8 wall-clock hours — but that includes nights, work, and life. Real time-on-task was not measured during this window, so I won’t publish an “hours saved” claim. (The instrumentation now records genuine active time — more on that at the end.)

The Model Policy, as Measured

I wrote the model policy down before starting, which makes it checkable afterwards. P-1, dated July 14: Sonnet as the implementation default, Fable 5 for planning and story creation, escalation for genuinely hard stories — change at most once mid-PoC.

The telemetry shows what actually happened: Sonnet led implementation for exactly three stories. From 1.4, Fable 5 led; from 1.6 through 2.5, Opus 4.8 exclusively; from 2.7 on, Opus 5 exclusively. That’s at least two changes, not one — driven by an amendment the second story forced (a Sonnet-created crypto story missed a hole in the cryptographic strategy that the stronger model caught, so crypto-touching story creation was escalated permanently), by weekly rate limits (story 1.5’s final fixes were demoted mid-task when a limit hit), and by Opus 5 simply being released mid-project. A policy remembered would say “we mostly followed it.” The policy as measured says the factory re-decided its staffing three times in three weeks, each time for a recorded reason.

The strangest entry is the one the operator didn’t decide. During story 1.2, safety routing demoted Fable 5 to Opus 4.8 mid-task because the work touched cryptography — the factory’s model routing is not fully operator-controlled, and the substitution was visible only because the telemetry carries a model label. That single data point became a standing constraint seven stories later: crypto-touching stories declare their model up front and record what actually ran, and a substitution is a finding to raise, not a routine event. Seven consecutive crypto-heavy stories then ran with zero substitutions. For any regulated environment that plans to adopt agentic pipelines, this is the governance-relevant fact in this post: the question “which model produced this artifact?” must be answered by telemetry, not by the agent’s own claim — although, for the record, the two agreed in every measurable case here.

Cache Economics

The token mix is the most lopsided distribution in the whole dataset:

Token type	Tokens	Share
Cache reads	2,197M	95.3%
Cache writes	92M	4.0%
Output (the writing)	10.0M	0.4%
Fresh input	1.4M	0.06%

Of 2.3 billion tokens, only ten million were the model actually writing — code, plans, review findings. Everything else was reading and re-reading its own context: the story file, the architecture docs, the accumulating conversation. And 95% of that reading was served from prompt cache, which is billed at roughly a tenth of the fresh-input price.

That discount is not an optimization detail; it’s the reason the model of operation works at all. An agentic dev loop re-sends its entire working context on every tool call — that’s what makes the agent coherent across a 13-session story. Billed as fresh input, that pattern would be ruinous; cached, the marginal cost of the loop’s next step is close to just its new tokens. The factory’s economics are cache economics: the bill scales with what the agent writes and decides, not with what it must keep in mind. It also explains why cost tracks scope rather than verification — a bigger story means more context held longer, and the meter mostly counts remembering.

Regenerate This

Every number above comes from one of two query idioms, and confusing them is exactly how the $60.80 happened:

1. Cumulative counters (cost, tokens): export the raw series, split into sessions at every counter decrease, sum each session’s last value. The quick approximation — good to ~1%, fine for checking, not for publishing:

sum by (issue) (increase(claude_code_cost_usage{project="pwa-wotlk"}[90d]))

2. Event samples (oracle runs, register sizes): last_over_time / count_over_time with an explicit window — never increase(), and never a bare instant selector, which silently misses anything older than a few minutes.

The full queries.md — including the export commands, the reset-aware aggregation script, and the correction log of the queries that were wrong first — ships with the repo when it goes public.

One last measurement note, because this series is live documentation: on August 2, the day of this snapshot, the factory’s instrumentation was overhauled. Cost and token counters now carry a session label (the reconstruction this post describes is no longer necessary going forward), and new gauges record genuine time-on-task, lines of code, and test-suite health. None of it backfills — everything in this post was derived the hard way, and the posts covering epics 4 and beyond will get to cite their numbers directly. When a chart in this series spans that boundary, it will say so.

Final Thoughts

The bill says a solo operator can ship 20 verified production stories for about $92 of metered compute each — or a tenth of that on a subscription. But the finding I’d actually forward to whoever owns your engineering budget is the other one: the measurement layer needs the same adversarial treatment as the code. My cost dashboard under-reported by 2.6× while looking perfectly healthy, and my two headline quality gauges spent the project reporting a tautology. Both were only caught because publishing forced regeneration from raw data. If your agentic pipeline reports its own health, ask when you last saw those numbers move.

Next installment, in two weeks: how the entire planning corpus — brief, PRD, UX, architecture, epics, 41 stories — was produced for $177.80, and what an adversarial review pass catches that a human reader doesn’t.