I Told Codex to Test Everything. It Believed Me.

When I finally had Claude go through three weeks of Codex's hardening work, its summary was blunt enough that I'm only paraphrasing a little: I was hardening a children's book website against NSA-level access by foreign threat actors.

Fair. The site is Cappy West, home of a 13-book series about a capybara with no sense of direction. It already existed before any of this started. Forty routes, five games, links out to retailers. No accounts, no forms, no uploads, no ads, no API. It's a static export served from a read-only, unprivileged Nginx container behind a Cloudflare Tunnel, and the only state it has, a child's game progress, lives on the child's own device.

By the time I made the stop official on October 21, Codex had spawned 443 subagents, run two complete 247-case crash-and-recovery campaigns, and designed a build pipeline that couldn't finish without Apple's permission. No hardened release ever reached the live site. A lot of that traces back to one sentence I typed on day one.

On September 6 I pointed OpenAI's Codex at my dev server with Daybreak Blue turned on. Daybreak is OpenAI's Trusted Access for Cyber program, and Blue is its option for authorized defensive work: flagship models with fewer refusals on security tasks. Its API alias at the time, gpt-daybreak-blue-latest, mapped to GPT-5.6 Sol. I asked for a read-only assessment. Test the site, report what you find, change nothing until I say so.

The audit agents started at 15:52 UTC. The report landed at 16:09. Seventeen minutes.

No Blocker. No High. Three Medium host-hardening findings: unrestricted passwordless sudo on the admin SSH account, password SSH with forwarding and X11 still enabled, and the tunnel daemon running as root with its full capability set. Six Lows, mostly cleanup. All 40 routes passed. TLS 1.0 and 1.1 were rejected. 225 tests green, zero npm audit findings, and a clean secret scan across 316 Git revisions.

That looked to me like an afternoon of fixes. I approved remediation, kept the sudo change for myself, confirmed I had a VM snapshot and a full backup from the night before, and typed: "Check whatever is needed and continue with hardening." Then I asked it to make sure the site still worked when it was done.

That sentence has no ceiling, and it leaves the agent to decide what "needed" means. I handed it to something that doesn't get tired or bored, and it took me at my word.

Here's what "whatever is needed" turned into.

Every deploy had to be bound to an exact commit, tree hash, container image, and policy set. A rollback and recovery controller got built, frozen, and rebuilt. Independent audit agents reviewed each other's work. One adversarial reviewer ran for over ten hours and found that a recovery process rejected the very sandbox mounts its own systemd unit created, a mismatch that passing tests built on fake mount inventories had hidden. Proposals and approvals got version numbers, and the version numbers kept climbing until they hit P51.

The validation VM is where the calendar went. Each test case ran the same ritual: revert the VM to an exact baseline snapshot, prove it was actually pristine, wait for a quiet window where no systemd timer was about to fire, import the exact bundle, then deliberately kill the process at one specific moment, mid-sync or mid-rename or mid-publish, and check that recovery put everything back where it belonged. Export the evidence, revert, and start the next case. 247 of them, strictly one at a time, because a revert wipes whatever the last case left behind.

If the timers wouldn't go quiet, the controller waited. If they still wouldn't, it refused to count the case. One later attempt burned through 400 readiness probes without finding a clean 270-second window and stopped before case 25.

The first full campaign took about 41 hours and passed 247 of 247. Then the next gate, installing builder authority, failed its preflight before touching anything. A corrected lineage ran a fresh campaign: about 50 hours, 247 of 247. Then builder authority failed again, this time because the checkout kept stricter file modes than the next step expected and a validation check tripped over how systemd reports an inactive timer.

Two perfect runs, about 91 hours of wall clock between them, and neither one unlocked a release.

That was roughly the first two weeks. The third happened mostly on my Mac, where the problems stopped being things a test suite could argue with.

One protected-build step failed because macOS stamped an 11-byte com.apple.provenance extended attribute onto files the policy said had to carry zero. Codex stopped and preserved the failure instead of waving it through, and I later approved a narrowly scoped correction. Correct call. Another revision.

GPT-6 Sol started rolling out to Codex on September 22. Two days later I switched to it, because it was newer and I figured newer meant better. Daybreak stayed on. Over the next two and a half days it spawned 259 subagents, 161 of them on September 26 alone. The previous 18 days had used 184, nearly all on that Blue alias.

That burst lines up with the last two proposal rounds, P50 and the P51 design. I can't tell you the new model caused it. I can tell you I gave up any way of finding out.

P51 wanted the Mac to prove that during a protected build, only approved files were ever opened, and that the guarantee held even if the process doing the watching died. The design Codex picked for that runs on Apple's Endpoint Security framework, and an Endpoint Security client needs a restricted entitlement that only Apple can grant. You submit a request and wait. I submitted one. As of this writing, Apple hasn't answered. Apple doesn't publish a timeline, either. When a developer asked on Apple's forums in August, an Apple support engineer replied that he wasn't involved in approvals and couldn't "offer any insight into the timeline." Other developers there describe waits of several months, and one reported a grant that took 13 months.

It also wanted a one-use external claim ledger on Cloudflare R2 so a build approval could never be replayed. That part was just more work. Apple approval wouldn't have finished the job, either: the integrated controller and a fresh 247-case run were still ahead. But the entitlement was the one piece I couldn't build my way past, and under that design there was no path forward without it.

At the last active checkpoint, September 27, P51 was an unfrozen draft with zero of its required 247 validation cases run. It sat there. On October 21 I stopped the scheduled continuation and made the stall official.

Meanwhile, one bug a visitor could actually trip over stayed put the whole time. Ask for /explore without the trailing slash and you got bounced to plain HTTP on port 8080. On the September 27 check, all 39 slashless routes did it. The likely cause is Nginx building an absolute redirect from what it sees inside the tunnel: plain HTTP on the container's port. Nginx's docs give the switch that makes those redirects relative:

absolute_redirect off;

Three weeks of crash-recovery evidence, and that fix never shipped. The record doesn't show the three Medium findings from day one closed, either. Seventeen minutes to find the problems. Three weeks later, none of them were confirmed fixed.

I keep wanting to blame the agent for this, and it doesn't hold up. Codex did what I said. I said check whatever is needed. I never said against whom, or how much was enough. A static book site needs protection from opportunistic scanners and automated exploitation. It doesn't need protection from a nation-state. I knew that. I didn't write it down, so as far as the agent was concerned, it wasn't true.

Everything else I learned hangs off that.

The Apple entitlement should have been the first thing the design surfaced, not the last thing it hit. It showed up about three weeks in. Next time, before an agent designs around any capability, it lists every approval that has to come from outside the room -- Apple, a cloud provider, a registrar, another team -- along with the expected lead time. Anything without a guaranteed turnaround gets designed around, or it kills the approach on day one.

The cheap, visible fixes go live first. By my estimate, the two Mediums I'd left to Codex, plus the redirect, were an afternoon of work. Instead they got folded into a release process that never shipped, so three weeks in, no hardened release had reached the live site. Real fixes should reach real visitors before the ambitious pipeline gets to chase anything.

Every revision resets the evidence clock, so decide up front what that's worth. Under the standard Codex was holding itself to, proof for one exact artifact doesn't count for the next one. That's the right rule. It's also why two perfect 247-case campaigns couldn't count toward the next release, and why the 143 passes from an earlier attempt stopped counting the moment it died on a four-file inventory check that expected a different sort order than find produced. The old results still told us things. They just couldn't be counted. I'd now set the number of full re-runs a project is worth before it starts. When it hits that number, it stops and asks.

"Unattended" doesn't take you out of the loop. It moves you to the approval queue. I checked in a few times a week, and most check-ins were sign-offs: confirm the snapshot, build the validation VM, authorize a rollback path scoped to one VM and one snapshot, loosen a Mac permission, approve a provenance correction, pick who would hold the keys on the R2 side. The status reports were accurate and written for an auditor, dense with commit hashes, proposal numbers, and finding IDs like P51-REPLAY-LEDGER-001. What I needed was three plain sentences a day: what changed, what's blocking, what you need from me.

And pick the model before you start. Switching mid-run means that when behavior changes, you can't tell whether it was the model or the problem. If something better ships halfway through, finish the phase you're in and switch at a boundary where you can compare.

Here's roughly what the first paragraph of my next security prompt looks like:

Scope: static site. No accounts, forms, or uploads.
Visitors submit no personal data.
Threat model: opportunistic scanners and automated exploitation.
Not targeted attackers, not nation-states.
Done means: the findings you report are fixed on the live host,
every public route is verified, and you stop.
Before depending on anything that needs outside approval,
stop and ask me.
Every day: three sentences, plain English.

The agent made real mistakes along the way, and it caught a lot of them itself. What it never heard was the word "enough," because I never said it.