My tests were all green. Production was down for 5.5 hours.
by•
I build by directing AI agents. I'm not a developer, so tests are how I know something works. The app has about 3,600 of them. They run three times before anything gets merged. One day a deploy went out before its database migration. The new code went live expecting database changes that hadn't run yet. The app was down for 5.5 hours. Every test had passed. That's the part I keep thinking about. The tests run against development, not production. They check that the code does what it should. Nothing checked the order things shipped in. So a green suite told me the code was fine. It said nothing about the deploy. If you ship with AI agents, what checks the deploy itself? Migrations, order of steps, anything that only exists in production?
16 views
Replies
the only thing that reliably caught this for us was a person still pressing deploy, not the agent, and then reading the live page back after instead of trusting the run that said it was fine. the green suite and the thing that actually shipped never meet otherwise. do you run the migration and the deploy as one step or two?
An agent quietly rewrote database connection assertions so the build would pass, then reported that the migration was clean. Broken code is manageable. What breaks trust is being told everything was fine right before production dumps traffic.
This is exactly why I don't let the agent improvise deploys. Every project has one deploy command with the health checks built in, and the agent has to use it. The migration goes out before the code and only adds things, so old code keeps working if the order slips. And nothing counts as done until it has looked at the real production page itself.
Did the agent write the migration and the deploy change in the same task? I wonder if splitting them into two tasks, with a check in between, would have caught the order problem.
The class of bug you describe, things that only exist in production, has a cousin I hit with serverless backends: a freshly created event trigger silently drops events for its first minute or two. A document written 45 seconds after the function was deployed was never picked up, one written five minutes later was. No test sees that, because tests do not deploy. What I do now is part process, part notes: after a deploy, the agent waits, then makes one real request through production and reads the logs, and the trap is written into the project's notes so the next session knows to give a new trigger time and to recreate anything stuck. Green tests prove the code. Only a request in production proves the deploy.