What's your actual process for catching bugs an AI coding tool introduces before they ship?
Been building almost entirely with AI-assisted tools, and the thing I've found matters most isn't how fast the AI generates code, it's having a real, repeatable way to verify what it actually did before trusting it's correct.
A recent example: I had a security researcher report what looked like a real vulnerability. Turned out the AI-assisted fix from weeks earlier had shipped correctly, but the researcher's finding conflated two different things, the anon key being visible client-side (expected, normal) versus whether it actually granted unauthorized data access (it didn't, verified by literally running real requests against the live endpoint). Took actual empirical testing to tell the difference, not just reading the code and assuming it was fine.
Curious what others have landed on for verifying AI-generated work before it ships. Do you have an actual process, or is it more ad hoc? What's caught you off guard that a quick glance wouldn't have?
Replies
The rule that made it repeatable for me: the tool's own report of what it did is not evidence. It describes intent, not effect.
I measured this on my own browser automation this month. Roughly one in three actions that reported success had silently done nothing. Every report read clean. The only check that caught them was reading the real state back afterwards.
Your live request test is the same move. The code says what was written. Only the state says what it does. What caught me off guard is that a false success reads exactly like a real one until you look at what actually changed.