3 silent failures in an AI app that threw no errors (and how I found them)
Three bugs from a solo-built AI app (Next.js, Supabase, Groq). None threw an error, and all of them looked fine in the logs.
Row-level security hid failed writes. Missing policies meant some writes quietly did nothing, and my code never checked. (With RLS on, an UPDATE that no policy allows touches zero rows and returns no error.) Fix: add the missing policies, and read important rows back to confirm the write happened.
A payment webhook returned 200 and upgraded nobody. The webhook arrives with no login cookie, so under RLS the update matched zero rows. Fix: use a server-side service client for that one route, verify a secret token first, and treat “0 rows updated” as a failure, not a success.
Two code paths overwrote each other. Achievements were unlocked from three places with no coordination, so one request could undo another’s write. Fix: a compare-and-swap write, which only updates the row if it still holds the value I read.
The common thread: a response with no error doesn’t prove anything happened. Now I verify the write itself.
What’s your go-to way to catch silent database failures?
Replies
Silent database failures are scary because everything looks fine in the logs but the data is wrong
Tracking down failures brings back memories of handling raw SQL scripts on old physical servers in the early two thousands. We would print out transaction logs, on dot matrix paper just to manually trace where data went missing during bulk inserts. Modern cloud infrastructure definitely hides those low-level details.
The webhook one is the bug I'm most afraid of. In my app the payments webhook can only credit a purchase that arrives with our user id, so the app has to tell the payments SDK who the user is before any paywall can open. Get that order wrong and the money comes in, the webhook says 200, and nobody gets anything. So an event without a user id isn't dropped: it's parked and logged as an error, and gets credited once we know whose it is. Same instinct as your "0 rows updated is a failure". Do you have anything that alerts you on those now, or do you still find them by reading logs?
Yep, "no error" doesn't mean anything happened. I gave up on reviewing every line a while ago, I lean on end-to-end tests that click through the real app. The agent also can't tell me it's done until lint, build and tests pass and it's actually opened the running app. And for the stuff tests won't catch, I keep an eye on API usage and my phone pings me when it goes over a threshold.