FetchSandbox
API integration testing that remembers what breaks
Start new thread

What changed since our last launch and why we trust it now!!!!!

by
•

Since our last launch we've been heads-down on one thing: making "this integration works" actually mean something. Here's what changed, and why we're more confident than a month ago.

What we shipped?

Five integrations now recognize the bugs people actually hit and prove the fix with a real before/after:

Stripe, Paddle, Clerk, AgentMail, Descope. Not "we support the API," but "we catch the specific way it breaks."

Chaos-order proof for billing webhooks. One of you asked whether we could prove a Paddle subscription survives webhooks arriving in any order. We built it, and then went deeper, to the failure devs actually fear: a paused customer silently getting their access back. We fire every delivery ordering (plus duplicates) at your handler, prove whether a stale event can re-entitle a canceled/paused subscription, and reduce the failure to the two events that cause it, e.g. "deliver paused, then a stale activated." Not "send a webhook." Reproduce the ugly delivery conditions that break production billing.

Receipts that read like a doctor's note: symptom, cause, fix, outcome, plus the real measured before/after. Not a green checkmark you take on faith.

Why we're confident (the part that matters)?

We tested recognition against real developer questions, harvested from GitHub/Stack Overflow, phrased by strangers, not us. When recognition was thin, the numbers said so (one integration went from catching 1 of 28 real phrasings to 22).

We asked a second model to poke holes in our test coverage, then verified every suggestion against the real engine before trusting it. It surfaced 5 false positives we'd otherwise have shipped.

We spent days trying to break our own work: malformed inputs, adversarial routing, the proof path itself. Every real bug we found (including two inside the chaos-order feature, caught by proving it on prod) we fixed before shipping.

What's still rough (because you'll ask)?

A couple of integrations we can't measure yet, the public bug corpus is too noisy. We'd rather say that than fake a number.

Proof against your actual code (via a Docker provisioner) works end-to-end but isn't in the default flow yet.

Want to kick the tires? Small apps — some with a real bug planted, no hints — you point your AI agent at to see if it catches it. ~20 min, no accounts or keys.

I want the "this didn't work" reports as much as the wins. Ask me anything.

53 views

Add a comment

Replies

Best

The chaos-order webhook testing is a nice touch. Those are exactly the kinds of issues that are hard to reproduce consistently. I'm curious: as APIs evolve, how do you ensure your bug signatures and proofs stay up to date without creating a large maintenance burden?

 good question man.. and it's the part I am most wary of overselling, so let me be straight.

two things keep it from being pure hand-maintenance.

1/the chaos proofs are written against invariants, not payload snapshots ..which means "a canceled sub can't reactivate," "a retried event can't charge twice."

those stay true no matter how Paddle versions its webhook body, so they don't churn when the API evolves.

2/the structural surface (endpoints, fields, enums) comes from the OpenAPI spec, so a re-import picks up most shape changes for free.

where it's honestly still manual: catching when a provider quietly changes behavior underneath you. the real fix for that is drift detection .. flag "reality moved, here's a new pinned version, review the diff, opt in".. and

that's on the roadmap for august.., not shipped.

so today it's the spec + the invariants doing the heavy lifting, and I watch the drift piece by hand.

rather tell you that than pretend it's automated.

 that distinction is the actual work, and i don't think there's a shortcut around it. the check i run when the adversarial model flags something i would have passed: does its reasoning survive a rename? if i swap the field name or endpoint path and the flag would evaporate, that's a pattern-match, not a catch. a real semantic bug survives surface changes because it's pointing at a trigger-behavior relationship, not a string.

in practice that means reading the adversarial model's reasoning against the scenario's trigger conditions, not just its verdict. the ones that are real almost always have a specific: "if the webhook fires twice before the ack, the handler will process the second delivery as new state." the ones that are noise say something like "this looks like it could fail" with no concrete trigger. the former i log as a calibration gap and update the rubric. the latter i discard and note why, because if i don't track the false positive shape i'll just see the same pattern-match again on the next run.

the harder case is when the adversarial model is partially right, it found a real failure mode but via the wrong reasoning. i've learned to treat those as real catches anyway and just correct the reasoning, because the scenario validity criteria i care about is whether it maps to prod behavior, not whether the model got there cleanly.

the 1-of-28 to 22-of-28 number is the kind of before/after that actually means something, most "we improved accuracy" posts don't show you the ugly starting point. having a second model red-team your own test coverage and catching 5 false positives you'd have shipped is a good process too - curious whether that adversarial pass is now just a permanent step before every release, or something you run occasionally when a feature feels risky.

appreciate that you called out the ugly starting number, that was a deliberate choice. on the adversarial pass: after it caught those 5 false positives i'd have shipped, it's hard to justify running it only occasionally. the goal is permanent, every release. the tricky part is keeping the red-team prompts calibrated as the underlying behavior shifts, so right now it's permanent but with a manual review step to make sure the adversarial model is still asking hard questions and not just passing things through on vibes.

 appreciate that you noticed the starting number and not just the end state, that framing was deliberate. on the adversarial pass: it's a permanent gate now, not occasional. after it caught 5 false positives i'd have shipped with confidence, running it only on "risky" features felt like the wrong call. the harder ongoing problem is keeping the red-team prompts calibrated as the underlying behavior shifts, so right now it's mandatory but with a manual review step to make sure the adversarial model is still asking hard questions and not just rubber-stamping.

the honesty about what's still rough is the part that actually builds trust here. the docker-provisioner real-app proof being end-to-end but not in the default flow yet is the detail I'd push on: is that a trust problem (running arbitrary code from someone's repo) or a cost/speed problem (spinning up containers per verification), because those have pretty different timelines to fix. until it's default, the receipts are still proving the fix against your scenario spec, not against the code that's actually going to ship, which is a meaningfully different claim than it sounds like at first read.

 thanks again man.. you are the one i always look forward for the comments on my post...and you have put your finger on the most important caveat in the whole post, and you're right on both the times..

which problem is it-- mostly trust, cost second. the provisioner runs your actual repo code in a container, and it does work end-to-end .. I hve put real repos through it and gotten a before/after on the shipping code, not just the scenario. but making that the default means running untrusted code from any repo we're handed, safely, at scale. that's an isolation surface I'm not willing to rush. the per-run container cost is real too, but it's the smaller problem with the clearer path.

aand the second point is the one I most want to be straight about..until the provisioner is default, the receipt proves the fix resolves the reproduced failure against our model of the API .. not that your exact shipping code passed. those are different claims, and you're right that the second reads like the first at a glance. the scenario-level proof is real (it reproduces the actual failure and shows it gone), but "proven against the code that ships" is the stronger thing .. it's opt-in today, not default. closing that gap is the roadmap item I care about most.

the manual review step on the adversarial model itself is the part most people would skip, it's easy to assume the red-team prompts stay sharp forever once you've written them once. how do you actually notice when they've gone stale though, is it a scheduled recheck or does it usually take a false negative slipping through to notice the calibration drifted

 honestly? closer to the second than I wud like there is no scheduled is the red-team still sharp recheck... but the shape of that failure is what makes it livable.,, th stale adversarial prompt can only cause a miss, never a wrong answer. because the deterministic engine is the arbiter, the red-team going dull shows up as under-coverage .. we didnot think to test that ordering ... not as a false proof shipping. the risk is a gap, not a lie, which is a much more forgiving failure mode to be stuck with.

how the gaps actually surface.. two live inputs instead of a frozen prompt set... the adversarial cases are seeded from real developer failures..so the wild keeps feeding it new phrasings and classes. and the blind playground + tester program is .. honestly the deliberate false-negative detector.... real people pointing agents at planted bugs surface what my red-team didn't think of, before prod does. a genuinely novel failure class still slips until someone hits it, which is exactly why the "tell me when it didn't work" ask is load-bearing, not politeness.

@Raj Nagulapalle mandatory-but-reviewed is a good middle ground, better than either "always trust the red team" or "only run it when we remember to." out of curiosity, who does that calibration review - is it you personally reading through what the adversarial model is asking, or is there a second automated layer checking that the red-team prompts haven't gone stale?

 it's me, personally .. I read what the adversarial model surfaces and decide what is a real class vs noise. no second automated layer watching the red-team for staleness.

and that's intentional.., not just small-team reality (though it's that too): an automated checker on the red-team would just push the calibration problem up a level .. who checks that? the reason it doesn't recurse forever is the deterministic engine. it isn't another opinion, it's ground truth .. a candidate either reproduces or it doesn't. so the loop bottoms out at human judgment for the hypotheses, deterministic reproduction for what's actually true.

the honest limit: that's one person's judgment today, and it has to become more than mine as this grows. I'd rather call that a known gap than bolt on an automated layer that feels rigorous and isn't.

'not we support the API but we catch the specific way it breaks' is the entire pitch. thats a claim thats falsifiable by the customer in 30 seconds. the industry defaults to 'we integrate with X' which is unfalsifiable and therefore uninteresting to a buyer who has been burned. the before/after receipt is the trust unlock. every dev tools company shipping an update should be doing this instead of 'were now compatible with Y.' congrats on the ship raj, going to run it against a paddle sandbox and report what surfaces.

 "falsifiable in 30 seconds" I am stealing that, it's the sharpest anyone's put it, me included. you nailed why "we integrate with X" is dead copy to anyone who's been burned: there's nothing to check...

the whole bet is that a before/after you can falsify yourself beats any claim you have to take on trust.

please do run it against Paddle .. and go for the mean case: fire the subscription events out of order and stale, and see if it catches a paused customer getting silently re-entitled. that's the one I would hand a skeptic to try to break. and I meant the ask in the post .. if something's weak or won't reproduce, that report is worth more to me than the wins. thank you, genuinely

@rnagulapalle orthogonal failure as the sharper metric than raw catch-rate makes sense to me. one thing I'd want in that rubric though - when the adversarial model flags something you'd have passed, do you check whether its stated reasoning is actually why it's a bug, or just that it landed on a real one for an unrelated or wrong reason? a model that's right for the wrong reason looks identical to a well-calibrated one in your log right up until the day it isn't.

 thanks again man.. you have found the real gap in the rubric, and no ...

today the engine checks the outcome .."does it reproduce", not whether the model's stated cause is the actual cause. right-for-the-wrong-reason passes today and you are 100% right that it looks identical in the log until the day it does not.

The one thing that keeps it from being fully blind the reproduction gets minimized bisected down to the fewest events that still trigger it. that minimized sequence is mechanical, not the model's opinion, so it's the ground truth "why." which means I can catch right-for-wrong-reason ... the diff the model's stated cause against the minimized cause, and a divergence is exactly the miscalibration you're describing.. thnks again..

what I don't do yet is make that diff an explicit gate item and assert "stated reason == reproduced reason," not just "it reproduced." i beleive that's a real addition and you basically just wrote the spec for it... the oracle's already sitting there; I'm not checking the reasoning against it on purpose.

you keep landing on exactly the things I'm wrestling with... hahahaha mind if I pull this into DMs? been wanting to pick your brain properly.

great question and honestly the one i think about most. the short answer is the reproduce→prove engine carries most of the maintenance. when an API spec changes, we generate candidate scenarios first and run them through the sandbox to verify they produce the expected failure before they ever touch routing. so the signatures are proven, not hand-maintained. the harder part is knowing when a vendor quietly changes behavior without bumping their spec, which is why we shadow new patterns before promoting them. still early but that loop is holding up better than i expected.

@rnagulapalle the "who checks the checker" framing is the right place to stop, an infinite regress of automated calibration layers doesn't actually buy you more truth, it just hides where the judgment call lives. as this stops being one person's read on the red-team output, what does that actually look like in practice - a second person reviewing the same surfaces independently, or something more like a written rubric so the judgment isn't just tribal knowledge in your head?

 the rubric is probably the more honest step before any structural signal, because "second reviewer" isn't real yet as a solo founder. what i'm working toward is explicit criteria for what makes a red-team scenario valid: maps to a known failure mode, the API actually behaves that way in prod, the trigger is something real integration code would hit, written somewhere outside my head. the structural signal i find most interesting is orthogonal failure: when the adversarial model flags something i would have passed, that gap is real data about calibration drift. the reverse, where i catch something it missed, is where the rubric gets updated.