247 upvotes, 34 comments, finished second on a Sunday. The July launch got 362 and finished third. Better rank, smaller day. Both true.
The comments were worth more than the votes. A few things I didn't have good answers for, which is more useful than an upvote.
"almost any fix makes the error disappear, far fewer make the data right" is the whole thing, and it is the same shape as the problem we keep running into.
to answer your question: yes, and the worst one was not even an agent. we added an anthropic key and three things were wrong at once. opus rejects an explicit temperature outright, one haiku model id had been retired and returned 404, and our own code sent a temperature on every call. nothing failed in testing because nothing in testing actually called it. any customer who had selected opus would have had every single reply fail on the first try. we were offering an integration nobody had ever executed.
the related one scares me more. we ran eight models against a live pricing api and two of them read the wrong row of a price ladder that was sitting in their context. one quoted 39.00 for an order that costs 9.60, the other quoted 9.00. the 39.00 gets caught by anyone glancing at it. the 9.00 does not, and that is the one that reaches a customer.
so the thing i would want to know about the proof step: does it assert the response shape, or the actual values? a 200 with a plausible wrong body is the failure that survives every check we have tried.
FetchSandbox
@jernej_jan_kocica three failures at once and none of them visible until the key hit prod, that's exactly the shape that's hard to catch in any test that doesn't actually run the provider's validation. the retired model id returning 404 especially, that's the kind of thing you only find when something real is on the other end. glad this resonated, and that war story is going straight into how i explain the "why" of this thing.
@rnagulapalle Ok good, glad it was useful. I will try the MCP on our Stripe webhook path when I get time. The case I want to see is a 200 with a plausible wrong body, that is the one that gets through everything we have.
Meridian
FetchSandbox addresses a real weakness in AI-generated integrations: verifying actual end states rather than merely checking for successful responses. The stateful sandboxes, failure injection, and proof receipts make this especially valuable for testing billing and authentication workflows before deployment.
One optional improvement would be showing a side-by-side state diff in each receipt, highlighting the expected and actual values across the full workflow. That would make subtle failures, such as a plausible but incorrect response body, easier to detect during review. Congrats on the launch!
FetchSandbox
@adityaharish2002 that side-by-side diff idea is exactly right, right now the receipt shows the final state but subtle field-level drift across a multi-step workflow is easy to miss. adding a before/after diff per step is on the roadmap and your framing of it (expected vs actual across the full workflow, not just the last call) is actually the cleaner way to think about it. appreciate the specific callout.
FetchSandbox
@tehreem_fatima5 pact is great for locking down the contract between a known consumer and provider in CI. fetchsandbox is less about "did the schema match" and more about running the full lifecycle: stateful CRUD, webhook delivery, auth failures, rate limits, all from inside the agent's IDE session. for an LLM agent the rapid iteration piece matters a lot, it can reproduce a specific failure, fix the code, and get a proof receipt back rather than just validating shape.
Netlify
Testing and verifying AI integration fixes in a sandbox before deploying saves so much headache. Congrats on shipping!
FetchSandbox
@thisiskp_ thanks man!!..Yeah exactly, same bug, very different discovery method. Finding it before deploy means you fix it in your editor. Finding it after means a customer tells you about it at some inconvenient hour. That's the whole thing we're trying to shift.
FetchSandbox
@thisiskp_ Thanks! Honestly that pain is exactly why we built this. Catching a bad config or an edge case token error in sandbox vs in prod is a completely different experience.. one costs you sleep, the other costs you 10 minutes.