FetchSandbox
API integration testing that remembers what breaks
Start new thread

thanks for an unreal launch, now tell me what you're shipping

by
•

launch yesterday (july 12th) we told you all about .

today i'd rather hear about you.

first, thank you. for the upvotes, but honestly more for the questions/comments n the conversations about idempotency, out-of-order webhooks, deterministic replay, and what it actually takes to gate a merge were sharper than anything i'd written for myself. a bunch of you basically handed me my roadmap.

I built this because.. i kept watching ai agents write stripe and auth integrations that passed every test and then broke on the first real webhook. green ci, prod on fire. i wanted a layer that runs the full integration lifecycle before prod, so the duplicate event or the retry hitting stale state shows up now, not at 3am from a customer email.

so here's my question back to you: what are you wiring up right now, and can this help you ship it without the launch-week surprise?

if you're building solo, prompting your way through an app in cursor or claude code: connect fetchsandbox to your agent once, then just prompt in plain english, "test my stripe integration." it runs the real workflows, throws the failure cases on demand, and hands you back a receipt showing it either survived or exactly where it broke. no keys, no account, no staging wait. that's the whole loop.

if you're on a team

shipping real payment or auth integrations and want proof before merge: run the failure scenarios (duplicate delivery, retries, stale state), grab the receipt url, drop it in the PR. review stops being "looks good, merge" and starts being "here's the run that shows the async path actually held."

either way, tell me: what did it catch for you, and what would make it something you can't ship without? reply here or tell me straight, good or bad. i want the real signal.

to stay honest like i tried to be all thread: i answered every hard question including the stuff that's not built yet. deterministic out-of-order replay and version-locked pins you can gate a merge on are next. i'll make noise when they land so you can hold me to it.

free to try, no keys needed. thanks again for a launch i genuinely wasn't expecting.

85 views

Add a comment

Replies

Best

Building to help people find compatible and aligned cofounders, a problem that I've faced for the past 4 years :)

finding aligned cofounders after feeling that gap for 4 years is a real one .. good luck with foundinity

@raj thanks for the api testing thread. the conversation you had with andras actually helped me think through my own product's audit trail design.

shipping TAM network next month. it's a signature graph for professional work. people sign what they saw others actually ship. thinking of it as the human equivalent of fetchsandbox: catch the "passed the demo but broke in prod" problem, except for careers instead of code. green ci prod on fire = the resume passing every keyword screen then falling apart six weeks in.

 the "green CI, prod on fire" parallel for careers is genuinely the clearest way I've heard someone frame the credential problem. curious how TAM Network handles the timing side: does the signature capture when the work shipped, or just that it shipped? that sequence feels like it matters a lot for the "who actually did what" proof, same way webhook order matters for integration state.

+1 to Gal — retry→double-charge from idempotency-as-an-afterthought is the canonical one, and the receipt-in-the-PR is what finally gives it an artifact instead of "trust me, I tested it locally."

The beat I'd add from the agent side: the real unlock is when the agent itself runs those failure scenarios before it opens the PR — not a human remembering to click "replay duplicate delivery." An agent writing a Stripe or auth flow calls the duplicate/retry/stale-state suite, and only proposes the merge once the receipt is green. The reliability check moves inside the agent's loop instead of being bolted on after.

And extending Gal's stale-state question: the nastiest cases aren't "stale," they're ordering — a refund landing before its capture, a succeeded arriving after a cancel. Schema inference won't catch those; the shape is valid, the sequence is wrong. Which is exactly why deterministic out-of-order replay (you said it's next) is what turns this from a fuzzer into a real merge gate: reproduce the exact ordering that broke prod, not a random one.

 the ordering point is the sharper one and you're right that schema inference won't catch it, the shape is valid, the sequence is wrong, and that's exactly where the fuzzer framing breaks down. a fuzzer can surface it randomly; what you actually need is deterministic replay of the specific sequence that broke prod. that's the design target for out-of-order replay: not "something bad might happen," but "this exact ordering broke the handler, here's the receipt, here's the fix, here's proof it survived the same sequence."

the agent-in-the-loop framing is where i want this to go too. the MCP surface exists precisely so the agent can call the duplicate/retry/stale-state suite itself before proposing the merge, not a human remembering to run it. the receipt becomes the artifact the agent cites in the PR description, not a screenshot from a local run that nobody can reproduce. reliability check inside the loop, not bolted on after.

 That agent-in-the-loop framing is the piece I'd push on next — in payment-webhook land the sequence that breaks you is rarely reproducible on demand, because it depends on a third party's retry/ordering behavior (PSP webhooks especially), not just your own handler. So the 'exact sequence that broke prod' has to be captured once, live, and then replayable forever — which sounds like exactly the receipt you're describing. Does the sequence library let you feed in a real captured ordering from prod, or is it limited to the scenarios you've pre-modeled (duplicate, stale, out-of-order)? That's the gap between 'we thought of this failure mode' and 'this specific customer's traffic did this to us.' 🙏

 you've drawn the exact line, and I won't blur it: today it's the pre-modeled set (duplicate, stale, out-of-order), not "feed in the real ordering your PSP actually sent you."

but the two halves aren't equally far off. the replay engine can already replay an arbitrary specified sequence deterministically .. that's what the out-of-order work is: hand it an ordering and it reproduces that exact one, not a random one. so what's missing isn't the replay, it's the ingest: a path to capture a real sequence from your prod webhook logs and drop it into the library as a permanent scenario. "capture once live, replay forever," exactly like you said.

and you're right that's the real gap .. "we thought of this failure mode" covers the classes everyone hits, but "this specific customer's traffic did this to us" is the one that actually kept someone up at 2am. it's the direction I care most about and honestly the most interesting thing on the roadmap. not built yet, though .. rather tell you that than imply the capture path exists when it's still replay-from-modeled-scenarios.

  yeah, that split makes sense — the replay's the solved part, the capture is the hard bit. One thing I keep running into if you go capture-from-prod: real webhook logs aren't just an ordering, they're full of live PII and signed secrets. So before a captured sequence can become a saved scenario you'd have to scrub it, otherwise "replay forever" also means storing someone's card metadata forever. Are you thinking record-then-sanitize, or redacting right at ingest so the raw never gets stored?

right now it's webhook retries on a payment flow that's genuinely kept me up - a provider retries on timeout, we process it twice because idempotency key handling was an afterthought, customer gets charged twice, support ticket at 2am. the "receipt url in the PR" idea is what got me, that's the part every team I've worked with skips because there's no artifact to point to, it's just "trust me I tested it locally." would try this just for that alone. curious how it decides what "stale state" looks like for a system it's never seen before though - is that something you configure per integration or does it infer it from the API schema?

 the 2am support ticket from a double-charge is exactly the scenario the duplicate-webhook simulation was built for. on the stale state question: right now it's configured per integration rather than inferred from schema. for the stripe duplicate-webhook case, the scenario spec bakes in the precondition (payment already processed, idempotency key present) so replaying hits the same branch deterministically. schema inference is on the roadmap but I wanted the first version to be something you could trust, not something that guesses. the receipt-in-PR idea resonating makes sense to me, it's the only artifact that actually survives the "I tested it locally" gap.

 "something you could trust, not something that guesses" is a good way to put it, and honestly the right call for v1. baking the precondition into the scenario spec rather than inferring it means the test is deterministic and you can actually explain to someone why it failed, versus a schema-inferred version where the failure mode itself is one more thing to debug. I'd rather write ten scenario specs by hand than debug why the inference engine thought two events were duplicates when they weren't. good luck with the roadmap on that one.

 I'd rather write ten scenario specs by hand than debug why the inference engine thought two events were duplicates..screenshotting that for when I'm tempted to over-automate v2. thanks man, this whole thread sharpened it.

honestly this kind of follow-up post is rarer than it should be, most makers just move to the next launch. right now I'm mostly stitching together small AI agent workflows for side projects, nothing with real payment integrations yet, but the "green CI, prod on fire" line is exactly why I'd reach for something like this the day I do ship a Stripe webhook. bookmarking it for that day.

 appreciate that ... and honestly the follow-up posts are more fun than the launch itself. when you do ship that first Stripe webhook, it's a two-minute connect and one prompt ("test my stripe integration") . ping me and I'll make sure it catches the duplicate/out-of-order stuff on your setup. the day "green CI, prod on fire" stops being abstract is exactly when it earns its keep...

@rnagulapalle appreciate that, will take you up on it when it's time. good to know exactly which prompt to reach for instead of having to figure it out cold in the middle of a launch scramble.

 that's exactly the point .. nobody should be figuring out the ugly failure cases cold at 2am mid-launch. when you hit that moment, ping me directly and I'll make sure it catches the nasty ones on your actual setup, not just the demo. and thanks again for how much you've sharpened this along the way.

 on the side note .. Two things, and zero rush on either

  1. I'd love to trade notes sometime — the ephemeral/live-capture line feels close to what you're building at Genway.

  2. I'm running a small pre-seed for FetchSandbox, and you think about this space deeper than almost anyone — whenever you have 20 min, I'd value your honest read. No pitch, no urgency. Either way, appreciate you.