What Only Broke Live

The first cold visit to the app on a family phone went like this: tap Sign in, authenticate at Keycloak, land back on the sign-in screen with no error, no explanation. Tap Sign in again — silently signed in. Every test in the repository was green. Every gate from post 3 had passed honestly. The culprit turned out to be a service worker the project had seated but deliberately not yet configured: its default auto-update behavior reloaded the page to activate on first visit, discarding the in-flight OIDC exchange — after the app had already scrubbed the single-use authorization code from the URL. The second tap worked because by then the worker had settled.

Post 4 was about tests that lie. This one is about the defect class where honest tests can’t even ask the question: failures that exist only in the seam between your code and the platform it ships onto — service workers, CDN edges, auth walls, browser UI policy, deployment plumbing. Each of the four rules below was paid for by a defect invisible to the repository by construction, and every one was found the same way: a human walking the deployed product on a real device, with the trace recorder running.

(Series rules apply: mid-product snapshot, every claim regenerable from the project’s records — here that means captured network traces and measured response headers, not telemetry queries.)

Four Rules, Learned Live

Rule one: every subresource the browser fetches on your behalf has its own credentials mode — your app’s own fetches being correct proves nothing about them. The app sits behind Cloudflare Access, and one story’s headline acceptance criterion was “the app installs to the home screen.” The walk produced the manifest, the icons — and no install prompt. The capture explained why: browsers request <link rel="manifest"> without credentials unless the tag opts in (that’s the HTML spec, not an Access quirk), so Access answered the anonymous fetch with a cross-origin redirect to its login page, CORS blocked it, and Chrome received net::ERR_FAILED where it expected JSON — no manifest, no installability, nothing erroring anywhere the app could see. The trap is that it recovers on a warm session, so it fails exactly when the auth session needs interaction — the precise scenario the feature existed for. The fix was one line plus two driven-red guards; the rule covers manifests, prefetches and worker registrations alike, and an auth-gated origin breaks each one differently.

Rule two: measure at the edge, from a client, on the deployed thing — and when you remove a self-healing behavior, enumerate what it was quietly covering. The same story replaced the service worker’s unconditional self-update with an explicit update screen. The walk then measured what /sw.js actually looks like from a phone: the origin sends no Cache-Control at all, and Cloudflare’s edge adds its own — max-age=14400, cf-cache-status: HIT. Browsers deliberately bypass their own HTTP cache for a top-level service-worker script, but a browser bypassing its own cache does not bypass a CDN’s — so for up to four hours after every release, the update screen’s one action re-fetched the same stale worker it was trying to escape. The old design would have healed itself eventually; the new, more explicit one had removed exactly that — and every environmental assumption the self-healing covered became load-bearing the same day. The fix is edge-side, invisible to any repository test.

Rule three: an acceptance criterion written as “the platform will offer X” is a bet on someone else’s roadmap. Another AC read “the install prompt is available.” Chrome throttles or omits its automatic install banner by design — and suppresses it for a while after any earlier dismissal — so the install worked, but only through the browser’s ⋮ menu, one line away from a similarly-named non-install option. The AC was satisfied on the strict reading while the requirement’s actual promise (“a factory-fresh phone joins the family unassisted”) was not, and no amount of manifest correctness summons UI a vendor has decided not to show. Where a requirement promises an outcome, write the AC against the outcome and own the affordance in your own UI — platforms retire UI.

Rule four: verify the deployed artifact, not the deployment mechanism. Every walk rested on a shared assumption: merge to main, images republish, the updater converges, the phone tests the new build. Then the homelab side checked, and all three deployment lanes were pinned to a specific image hash from two stories earlier — and Watchtower cannot update a pinned hash; it skips it silently. A device walk run right after a merge would have exercised a three-story-old bundle while every participant believed otherwise. The tell had been sitting in plain sight: one live check kept failing run after run and was written off twice as “the fix isn’t deployed yet” without anyone asking why it wasn’t. A residual that keeps failing for an unexplained reason is a finding wearing a bug’s costume. The walk sheet now starts by comparing the served bundle against a captured baseline, because “latest” is not a fact about a deployment; it’s a claim about a config file somewhere else.

What Actually Catches These

None of these were found by a test, and none could have been — the failing component was the composition of the app with an edge cache, an auth wall, a browser policy, or a deployment config that lives, by deliberate design, in a different repository. What caught them was a practice the factory treats as a first-class verification lane, not as QA theater:

• The recorded device walk. Every Oracle-blind story ends with me walking the deployed product on a real family phone, cold state, against a written sheet, with findings recorded into the ledger. The walks above produced every specimen in this post. A demo shows the happy path; a walk hunts.
• Measure at the edge, from a client, on the deployed thing. The service-worker cache was invisible from the origin’s config and from the repo; one curl from outside the LAN settled it. Claims about headers, caches, and credentials are only facts at the point where a real client stands.
• Read the whole trace. The manifest failure was discovered by accident, sitting in a capture taken to diagnose the sign-in race. A trace aimed at one question answers others for free, and those answers are worth reading before the tab closes. (Its sibling lesson, learned the same night: a console with Preserve log on is not a log of the run you just did — two findings nearly got merged into one wrong conclusion because attempt 1’s errors were still on screen during attempt 2.)
• Ask the gateway for a clean “no”. Half the diagnostic pain in the manifest incident was inferring an auth failure through two layers of CORS wreckage. It turns out Access answers a request carrying X-Requested-With: XMLHttpRequest with a plain 401 instead of a redirect — so the app’s own session probe now gets a status code where it used to get a guess. Most gateways have a non-navigational mode; using it converts misattribution into a number.
• Interrogate residuals. The question that would have found the pinned deploy two stories earlier is the cheapest tool in this post: what would have to be true for this to keep failing?

Regenerate This

Documentary evidence again, but of a different kind than post 4’s: devtools network captures (the cold-open trace with its two redirects and cookies_sent: NONE), decoded Access tokens, and response headers measured from a client outside the LAN. All quoted in the story files and findings ledger (F-22, F-30 through F-34), which ship when the product repo goes public. The edge-cache measurement is a one-liner from any machine — request the service worker path and read cache-control, cf-cache-status, and age — and it’s the regeneration I’d actually encourage: run it against your own PWA.

Second Reading: September 4 — The Phone Is Also a Dispatch Surface

The phone on the couch turned out to be more than a test surface. Since early August, three home-screen buttons on an Android phone start or reattach the factory’s planning, development, review and QA sessions inside a terminal multiplexer on the laptop, over SSH on the LAN; the Claude mobile app then drives the session by remote control. Nothing about the compute moved to the phone — and that’s the design constraint, not a limitation: the sessions stay attributable, the telemetry export stays local, and the metrics store never has to be reachable from outside the network.

The design contains a defect story of the kind this post is about. The obvious way to stop a session is to kill every process by name — and while testing exactly that, a pkill -f pattern matched the very shell running it. The shipped stop button instead selects from a fixed allowlist of three session names, never a pattern, so a Claude session in a plain laptop terminal is not skipped but structurally unreachable: there is no code path that could select it. It sends the polite signal only, so transcripts stay resumable, and refuses to kill its own session. The register now carries a request for story-named sessions that would force the allowlist to become a pattern — and the item refuses to be sized until the safety property has been re-argued rather than assumed, because a pattern is one bad glob away from the command the script was written to avoid.

And the slow leak that only a phone loop reveals: a session ends only when you exit it. Detaching, closing the phone, dropping SSH, closing a terminal tab — none of them end a session, and each looks exactly like finishing. Measured on the laptop one morning: six live agent processes, about 3.3 GB resident, the oldest running for two days and three hours, half of them in forgotten terminal tabs. The property that makes the multiplexer worth having — survives everything that looks like an ending — is the same sentence read from the other side.

Final Thoughts

There’s a boundary around what a repository can verify, and this factory pushed that boundary about as far as it goes — an external oracle, three gate layers, guards driven red before their greens are believed. Everything in this post lived on the far side anyway. That’s not an argument against the gates; it’s the honest shape of the problem: the platform seam is a verification lane of its own, and it needs its own budget — real devices, cold state, traces read end to end, and the deployed artifact checked against what you think you shipped.

If your agentic pipeline’s definition of done ends at CI, these defects don’t get found later — they get found by users, one confused tap at a time, on exactly the day the demo matters.

Next and last in the series: translating all of it — deviation registers, gate declarations, ceremony budgets, procedure exceptions — into the language of organizations that answer to auditors. That’s Governance in Regulated Environments, in two weeks.