My agents wrote a webhook handler that passed review and CI. It still double-charged customers.
What happened
A few weeks ago my coding agent wrote a Stripe webhook handler. Signature check, event type check, fulfillment, clean 200. I approved it in 90 seconds because every line was correct.
And it was, until Stripe delivered the same event twice. Which it's allowed to do. Then the handler credited the customer twice.
Why it slipped through
The bug wasn't on any line. It was the assumption between the lines: every event arrives exactly once. Reading the diff can't catch that. My tests didn't either. I wrote the fixtures, so I never thought to send the same event twice. My assumptions were just checking my assumptions.
What actually caught it
Running the failure. Deliver the event, deliver it again, watch the double-credit, add the dedupe, run it again, watch it hold.
Agents probably make this worse because they learned from the same quickstarts that only imagined one clean event.
Curious what broke for you
A bug that sailed through review and CI but still blew up in prod, and what would have actually caught it?
(This is the itch I'm building FetchSandbox around, running the failure before customers do.)
Replies
the "my tests didn't catch it because I wrote the fixtures" line is the real bug, not the webhook itself. that's true of basically every test suite an agent or a human writes solo, you only test the failure modes you already imagined. the ones that get you are the ones nobody imagined, by definition. the closest thing that's worked for me is treating "what does the third party get to do that I'm not choosing" as its own checklist item during review, separate from does the code look correct. retries, out of order delivery, partial failures. none of that shows up in a diff, it only shows up if you go read the provider's actual guarantees (or lack of them) instead of assuming sane defaults
The part that lands for me is that your fixtures were written by the same head that wrote the handler, so the tests could only ever confirm the assumption instead of testing it. At least once delivery is documented behaviour, not an edge case, and every payment provider does it. The fix that actually holds is storing the event id with a unique constraint and letting the database reject the second write, rather than checking in application code, because two copies of the same event can arrive close enough together that both pass the check before either commits. Replaying the same event twice belongs in the test suite the way a null belongs in a form test.
gal and arnold both nailed the fix side, unique constraint at the write layer instead of an application check. the part I keep thinking about is detection, not prevention: how did you actually notice this in the first place, before a customer emailed you? was it a support ticket, a reconciliation job catching the mismatch, or did something alert on it automatically? for a bug that's silent by nature, the detection path feels like the harder problem to design for than the fix once you know what to look for.
The cheap version nobody's mentioned: run the suite twice.
Same fixtures, same events, replayed back to back in one run, then assert nothing changed the second time. Any handler that quietly assumes exactly-once fails immediately, and you never had to imagine the failure to catch it.
That's why it generalizes past webhooks. It's the same test for anything a provider is allowed to retry. Costs one CI flag.
The handler had no idea whether it had already run. That isn't a test gap, it's that the work had no identity.
Everything I build addresses a unit of work by run and step and writes its output under that key before the next one starts. "Have I done this" becomes a lookup instead of a judgment call. Nothing to remember, nothing for a reviewer to spot.
A replay test catches this one. Giving the work an address means the next twelve never happen.
+1 to Arnold's unique-constraint-at-the-write-layer, that's the version that survives the race. One thing I'd add: the constraint only works if the event id is still in your table when the duplicate lands, and you can't keep every id forever, so most people put a TTL on the idempotency store. Then the bug quietly moves to "duplicate arrives after the window I picked", Stripe can redeliver for up to 3 days, so if you prune at 24h the second charge sails through again. The dedupe window just becomes another unreviewed assumption, same shape as the original one.
Omri's detection question is the one I'd sit with longest though. For something silent by design, what actually caught it for us wasn't our own logs, it was reconciling against the provider's ledger (Stripe balance transactions) on a schedule and alerting on any mismatch. Your own event store can't catch a bug in how it processes events, you need a second source of truth that didn't run through the same code. Curious whether FetchSandbox's replay can fast-forward past a dedupe TTL, or if it's back-to-back redelivery for now?