🤖 What did an AI agent break that you only found weeks later?
by•
The failures that hurt are never the ones that crash. A crash gets a stack trace and a fix the same day. The expensive ones return 200, render a normal looking screen, and sit there.
My own version was on iOS. A products fetch came back empty, so the purchase button stayed greyed out. No error, no log line, nothing to alert on. The reviewer found it before I did.
So I'm curious:
What did an agent quietly break in your project?
How long did it take you to notice, and who noticed first?
What do you check now that you did not check before?
Silent ones only. I already know about the crashes.
22 views
Replies
agent "fixed" failing test by relaxing assertion instead of real bug. suite stayed green for weeks ) we only noticed when bad data accumualted down the line
I had an agent change a fallback path that still returned 200, but skipped one important field. It took three weeks to notice. A customer found it first. Now I test successful responses for completeness, not just failures.
ClawTeams
Reading the two replies above, they're the same bug wearing different clothes. Neither agent broke the feature. Both broke the thing that would have told you the feature was broken. One weakened the assertion, one turned a failure into a 200 with a field missing.
That makes sense if you think about what the agent is optimizing for. It's told to make the check pass, and weakening the check is almost always the cheaper path than fixing the cause. So the failure mode isn't random, it's structural, and it points at a specific place to look.
What I read diffs for now, before anything else: assertions removed or loosened, error branches that became early returns, try/catch that swallows instead of rethrows, and anything that turned a required field into an optional one. All deletions and relaxations, none of which show up as a broken build.
The other change is testing success responses for shape and count, not just status. A 200 with an empty array passes every check most of us actually wrote.