FetchSandbox
API integration testing that remembers what breaks
Start new thread

My agents wrote a webhook handler that passed review and CI. It still double-charged customers.

by
•

What happened

A few weeks ago my coding agent wrote a Stripe webhook handler. Signature check, event type check, fulfillment, clean 200. I approved it in 90 seconds because every line was correct.

And it was, until Stripe delivered the same event twice. Which it's allowed to do. Then the handler credited the customer twice.

Why it slipped through

The bug wasn't on any line. It was the assumption between the lines: every event arrives exactly once. Reading the diff can't catch that. My tests didn't either. I wrote the fixtures, so I never thought to send the same event twice. My assumptions were just checking my assumptions.

What actually caught it

Running the failure. Deliver the event, deliver it again, watch the double-credit, add the dedupe, run it again, watch it hold.

Agents probably make this worse because they learned from the same quickstarts that only imagined one clean event.

Curious what broke for you

A bug that sailed through review and CI but still blew up in prod, and what would have actually caught it?

(This is the itch I'm building FetchSandbox around, running the failure before customers do.)

49 views

Add a comment

Replies

Best

the "my tests didn't catch it because I wrote the fixtures" line is the real bug, not the webhook itself. that's true of basically every test suite an agent or a human writes solo, you only test the failure modes you already imagined. the ones that get you are the ones nobody imagined, by definition. the closest thing that's worked for me is treating "what does the third party get to do that I'm not choosing" as its own checklist item during review, separate from does the code look correct. retries, out of order delivery, partial failures. none of that shows up in a diff, it only shows up if you go read the provider's actual guarantees (or lack of them) instead of assuming sane defaults

 thanks again...that checklist framing is better than how i put it, "what does the third party get to do that i'm not choosing," held separate from "does the code look correct." the second question is the one everyone's trained to answer. the first is where the money leaks.

the part i keep bumping into: even once the checklist points you at "retries can happen," you're right back to reading a diff going "yeah, looks like it dedupes." the checklist tells you what to worry about, it still can't tell you whether the code actually survives it. reading the provider's guarantee and proving your handler holds under it are two different things, and the gap between them is exactly where mine shipped.

so i've started treating that checklist as the list of things to actually fire, not just review for. same instinct, one step furthe

The part that lands for me is that your fixtures were written by the same head that wrote the handler, so the tests could only ever confirm the assumption instead of testing it. At least once delivery is documented behaviour, not an edge case, and every payment provider does it. The fix that actually holds is storing the event id with a unique constraint and letting the database reject the second write, rather than checking in application code, because two copies of the same event can arrive close enough together that both pass the check before either commits. Replaying the same event twice belongs in the test suite the way a null belongs in a form test.

 thanks again..you're right, and it's a level below where my post stopped. the app-level dedupe i described still races: two copies of the same event land within milliseconds, both read 'not seen yet,' both write. looks correct in review, passes the single-delivery fixture, and still double-writes under the exact condition that triggers it, concurrent redelivery. the unique constraint on event id is the only version that actually holds, because there the check and the write are one atomic thing.

which is sort of the whole point. you can't tell the racy version from the correct one by reading either, both dedupe. you only find out by firing two copies at once and seeing whether the second bounces off the constraint or slips through.

and yeah, 'deliver the same event twice' should be as automatic in a webhook test as a null in a form test. the fact that it isn't yet is the gap.

gal and arnold both nailed the fix side, unique constraint at the write layer instead of an application check. the part I keep thinking about is detection, not prevention: how did you actually notice this in the first place, before a customer emailed you? was it a support ticket, a reconciliation job catching the mismatch, or did something alert on it automatically? for a bug that's silent by nature, the detection path feels like the harder problem to design for than the fix once you know what to look for.

 honestly.. we caught it pre-prod by running the duplicate intentionally. kind of a cheat, since it only works if you think to do it, and most of this thread is about how you usually don't.

for the ones that slip through, detection's tougher, and for the same reason: your logs and db agree with themselves. a double-credit just looks like two legit credits. only an independent truth source catches it, reconciling your ledger against the provider's record and alerting on mismatches. logs won't save you; they'll confirm the wrong number.

same shape as testing, really. you can't be your own oracle pre or post-deploy. before, you need something to fire the failure you didn't think of. after, you need something besides your db saying all's good.

+1 to Arnold's unique-constraint-at-the-write-layer, that's the version that survives the race. One thing I'd add: the constraint only works if the event id is still in your table when the duplicate lands, and you can't keep every id forever, so most people put a TTL on the idempotency store. Then the bug quietly moves to "duplicate arrives after the window I picked", Stripe can redeliver for up to 3 days, so if you prune at 24h the second charge sails through again. The dedupe window just becomes another unreviewed assumption, same shape as the original one.

Omri's detection question is the one I'd sit with longest though. For something silent by design, what actually caught it for us wasn't our own logs, it was reconciling against the provider's ledger (Stripe balance transactions) on a schedule and alerting on any mismatch. Your own event store can't catch a bug in how it processes events, you need a second source of truth that didn't run through the same code. Curious whether FetchSandbox's replay can fast-forward past a dedupe TTL, or if it's back-to-back redelivery for now?

 the TTL point is the sharpest thing in this thread. the dedupe window is just the original assumption wearing a different hat, you pick 24h, stripe redelivers for 3 days, and the second charge walks straight through the gap you didn't review. every fix here seems to plant one of these a layer down.

and yeah, you and omri landed on the thing i'd trust most: reconcile against the provider's ledger, not your own event store. a bug in how you process events can't be caught by the store those events ran through, you need a second source of truth that didn't touch your code. balance transactions are a good one.

straight answer on the replay: today it's back-to-back redelivery, the retry storm, which catches the race and the naive dedupe but not your TTL case. fast-forwarding past a dedupe window means controlling delivery time, not just order, and that's the axis i'm building toward next, precisely because the window is where the bug hides once you've fixed the obvious one. so honest no today, and it's exactly what i want it doing.

thank you for everytime i share some update you guys are helping like product owers and pushing the bar.. your feedback really really helping me harden the engine... this si amazing support guys for solo founder like me... who is waking up 4am grilling before going fulltime job hahaha .. idk just felt like thanking u alll

the independent truth source point is the real answer here, but doesn't that just push the problem up a level - now the reconciliation job is the thing you're trusting blindly. what stops that from silently breaking the same way, like a provider API change quietly making the reconciliation always report clean? feels like you eventually need a canary that deliberately injects a known mismatch to prove the checker is still watching

the provider-reconciliation idea holds up, but is the provider's record actually independent, or just a second read of the same upstream write path. stripe's events feed and their balance transactions api can disagree with each other during retries, so if your reconciliation job trusts whichever one it happens to query first you could still get a false-clean result. feels like the independence needs to come from a different write path entirely, not just a different read endpoint on the same system

the reconciliation point is the real answer and it matches what i've seen elsewhere too. the part that still bugs me is the lag - if you reconcile nightly, you find out you double charged someone up to 24h after it happened, which is fine for catching the bug but not great for the customer who already saw two charges hit their card. feels like the fix is layered: reconciliation to catch what slips through, plus something faster and cheaper checking the hot path so most of it never reaches that stage