What happened

A few weeks ago my coding agent wrote a Stripe webhook handler. Signature check, event type check, fulfillment, clean 200. I approved it in 90 seconds because every line was correct.

And it was, until Stripe delivered the same event twice. Which it's allowed to do. Then the handler credited the customer twice.

Why it slipped through

The gap most auto-fix tools have

They tell you the bug is fixed. They don't show you. Our rule has always been: make the bug happen on real code, apply the fix, show it stops happening. The gap was we could only do that for bugs we'd scripted in advance. Anything unusual and the honest answer was "found it, fixed it, can't prove this one."

What we built

We taught FetchSandbox to write the reproduction itself, no pre-scripted test required.

Since our last launch we've been heads-down on one thing: making "this integration works" actually mean something. Here's what changed, and why we're more confident than a month ago.

What we shipped?

Five integrations now recognize the bugs people actually hit and prove the fix with a real before/after:

launch yesterday (july 12th) we told you all about @FetchSandbox.

today i'd rather hear about you.

three bugs an ai wrote for me this week. every test green. all three would've paged me in prod:

1. webhook retried charged the customer twice

2. subscription.activated arrived before subscription.created they paid, app said not entitled

3. a stale event flipped a canceled sub back to active free access, forever

none of these show up in a 200. you find them from a customer or you fire the ugly event orders on purpose and find them first.

we now throw every ordering + every duplicate at your handler and hand back the exact sequence that broke it.

stripe sends webhooks at-least-once. so what's the actual risk?

one paid invoice, two deliveries, non-idempotent handler: you've just provisioned twice. you won't catch it in review because the code is correct on the first delivery. the bug only shows on the second.

so how do you even reproduce it?

I stopped reading and started running. took a real express and postgres billing app, replayed the scenario in a sandbox: same stripe webhook, delivered twice, like a real retry. no prod involved. the buggy handler provisioned seats 1, then 2, then 3. signature verification was clean throughout.

Most AI tools, when they're unsure, guess confidently.

That's how you burn an hour chasing the wrong bug.

We went the other way. Point ours at your repo and say what's breaking "customers are getting charged twice." Instead of guessing which API, it reads your code:

• Sees you have both Stripe and Paddle installed.

• Reads both webhook handlers your Paddle one has no idempotency check (line 16); Stripe dedupes fine.

• Tells you: "it's your Paddle handler." A fact from your code, not a hunch.

since the last launch, the goal was one thing: make FetchSandbox something you can actually gate a release on. not "looks tested" but provably tested.

what changed:

i build dev tools solo and use claude/cursor daily for API integration work. the pattern i keep seeing: agent finds a bug, fixes the code, declares done. what it does not do is rerun the exact failure scenario and prove the fix holds. that verification step was always on me.
we just shipped something in FetchSandbox to close that loop. after it catches a bug, it now nudges the agent to rerun the same request and surface a behavioral comparison, before vs after. same scenario, does it pass clean? that receipt is now automatic.
my rule: finding the bug is table stakes. proof that the behavior changed is the job. "i fixed the code" and "the fix works" are not the same claim.
curious how others handle this. do your agents ever rerun failure scenarios to verify, or does that step still fall back to you?