What changed since our last launch and why we trust it now!!!!!
Since our last launch we've been heads-down on one thing: making "this integration works" actually mean something. Here's what changed, and why we're more confident than a month ago.
What we shipped?
Five integrations now recognize the bugs people actually hit and prove the fix with a real before/after:
Stripe, Paddle, Clerk, AgentMail, Descope. Not "we support the API," but "we catch the specific way it breaks."
Chaos-order proof for billing webhooks. One of you asked whether we could prove a Paddle subscription survives webhooks arriving in any order. We built it, and then went deeper, to the failure devs actually fear: a paused customer silently getting their access back. We fire every delivery ordering (plus duplicates) at your handler, prove whether a stale event can re-entitle a canceled/paused subscription, and reduce the failure to the two events that cause it, e.g. "deliver paused, then a stale activated." Not "send a webhook." Reproduce the ugly delivery conditions that break production billing.
Receipts that read like a doctor's note: symptom, cause, fix, outcome, plus the real measured before/after. Not a green checkmark you take on faith.
Why we're confident (the part that matters)?
We tested recognition against real developer questions, harvested from GitHub/Stack Overflow, phrased by strangers, not us. When recognition was thin, the numbers said so (one integration went from catching 1 of 28 real phrasings to 22).
We asked a second model to poke holes in our test coverage, then verified every suggestion against the real engine before trusting it. It surfaced 5 false positives we'd otherwise have shipped.
We spent days trying to break our own work: malformed inputs, adversarial routing, the proof path itself. Every real bug we found (including two inside the chaos-order feature, caught by proving it on prod) we fixed before shipping.
What's still rough (because you'll ask)?
A couple of integrations we can't measure yet, the public bug corpus is too noisy. We'd rather say that than fake a number.
Proof against your actual code (via a Docker provisioner) works end-to-end but isn't in the default flow yet.
Want to kick the tires? Small apps — some with a real bug planted, no hints — you point your AI agent at to see if it catches it. ~20 min, no accounts or keys. https://github.com/fetchsandbox/playground
I want the "this didn't work" reports as much as the wins. Ask me anything.
Replies
The chaos-order webhook testing is a nice touch. Those are exactly the kinds of issues that are hard to reproduce consistently. I'm curious: as APIs evolve, how do you ensure your bug signatures and proofs stay up to date without creating a large maintenance burden?
the 1-of-28 to 22-of-28 number is the kind of before/after that actually means something, most "we improved accuracy" posts don't show you the ugly starting point. having a second model red-team your own test coverage and catching 5 false positives you'd have shipped is a good process too - curious whether that adversarial pass is now just a permanent step before every release, or something you run occasionally when a feature feels risky.
FetchSandbox
appreciate that you called out the ugly starting number, that was a deliberate choice. on the adversarial pass: after it caught those 5 false positives i'd have shipped, it's hard to justify running it only occasionally. the goal is permanent, every release. the tricky part is keeping the red-team prompts calibrated as the underlying behavior shifts, so right now it's permanent but with a manual review step to make sure the adversarial model is still asking hard questions and not just passing things through on vibes.
the honesty about what's still rough is the part that actually builds trust here. the docker-provisioner real-app proof being end-to-end but not in the default flow yet is the detail I'd push on: is that a trust problem (running arbitrary code from someone's repo) or a cost/speed problem (spinning up containers per verification), because those have pretty different timelines to fix. until it's default, the receipts are still proving the fix against your scenario spec, not against the code that's actually going to ship, which is a meaningfully different claim than it sounds like at first read.
FetchSandbox
great question and honestly the one i think about most. the short answer is the reproduce→prove engine carries most of the maintenance. when an API spec changes, we generate candidate scenarios first and run them through the sandbox to verify they produce the expected failure before they ever touch routing. so the signatures are proven, not hand-maintained. the harder part is knowing when a vendor quietly changes behavior without bumping their spec, which is why we shadow new patterns before promoting them. still early but that loop is holding up better than i expected.
the manual review step on the adversarial model itself is the part most people would skip, it's easy to assume the red-team prompts stay sharp forever once you've written them once. how do you actually notice when they've gone stale though, is it a scheduled recheck or does it usually take a false negative slipping through to notice the calibration drifted