We spent three weeks trying to break our own tool. Here's what changed
since the last launch, the goal was one thing: make FetchSandbox something you can actually gate a release on. not "looks tested" but provably tested.
what changed:
bulletproofed 5 integrations end to end: Stripe, Clerk, Descope, Resend, AgentMail. tested against how these APIs actually break in prod (webhook retries, out-of-order events, fresh-resource races), not made-up failures.
Paddle got the deepest treatment:vwe now throw webhook events at your billing handler out of order, duplicated, and stale, then hand back the exact sequence that broke it, trimmed to the fewest events that reproduce it. proves a paused subscription can't get silently reactivated, and a retried event can't charge twice. shipped: terminal-state safety, event-freshness checks, a typed "charge-once" invariant, minimized failing sequences.
got a second opinion: had another model poke holes in our coverage, then verified every case against the real engine. real-world phrasing recall went from 1-in-28 to 22-in-28.
tried to break our own work like a skeptical QA lead: cross-spec false positives: 0. same answer across 6 production runs. found and fixed two real bugs in the process.
every result explains itself now,
symptom, cause, fix, outcome, in plain English on the receipt.
next sunday (aug 2) we launch the version this was all for: every AI-written integration ships with proof it works. not a green checkmark you hope is right, a receipt you can put in front of your team.
coming up right after: more chaos mutations for Paddle (and the other specs), and the one we're most excited about, an agent that doesn't just find the bug but fixes it and proves the fix holds.
Paddle's our proving ground for it. 👀
Replies
the minimized failing sequence part is what stands out to me, most people just dump the full chaotic event log when something breaks and leave you to find the actual trigger yourself. trimming it down to the fewest events that reproduce the bug is the difference between a debugging session and an afternoon lost. did the second-opinion model actually find anything you'd missed, or mostly confirm what you already had covered?
FetchSandbox
@omri_ben_shoham1Â the minimization step was genuinely the hardest design call, most tooling stops at "here's the full event log, good luck," which just relocates the debugging work to you. trimming to the minimal reproducing sequence is what makes it actionable instead of archival.
on the second-opinion model: mostly confirming, honestly. it caught a couple of ordering permutations i hadn't explicitly modeled but had already covered through the deterministic engine. the more interesting thing it surfaced wasn't missing coverage so much as coverage i had right but couldn't articulate clearly before it named it. that second pass is useful less for discovery and more for making implicit assumptions explicit.
@rnagulapalle that distinction between discovery and making implicit assumptions explicit is a really useful way to frame it, and it matches what I'd expect from a good second pass. the failure mode I'd worry about is the opposite direction though, a reviewer that's confident enough to make you stop double checking your own model. does the second-opinion tool ever flag its own uncertainty, or does it always come back with a definitive-sounding answer either way