FetchSandbox
API integration testing that remembers what breaks
Start new thread

every test passed. i still shipped three bugs

by
•

three bugs an ai wrote for me this week. every test green. all three would've paged me in prod:

  1. webhook retried → charged the customer twice

  2. subscription.activated arrived before subscription.created → they paid, app said not entitled

  3. a stale event flipped a canceled sub back to active → free access, forever

none of these show up in a 200. you find them from a customer — or you fire the ugly event orders on purpose and find them first.

we now throw every ordering + every duplicate at your handler and hand back the exact sequence that broke it.

which of these has bitten you? (i've shipped #1 myself. be honest ...)

46 views

Add a comment

Replies

Best

the stale event flipping a canceled sub back to active is the one that'd actually scare me, that's not a bug you find from a support ticket, it's free access forever until someone happens to audit billing. out-of-order webhook delivery feels like the kind of thing every payments integration eventually hits but almost nobody writes a test for on day one because it doesn't happen until you're at real scale. did you find these three by throwing your own tool at your own webhook handler, or did a customer's setup surface them first

 found all three by running FetchSandbox against our own webhook handler before anyone else touched it. the stale event one specifically, you're right that it's the scariest category. a support ticket tells you something broke. free access silently granted never generates a ticket, it just leaks revenue until an audit catches it months later. that asymmetry is why i think the failure modes that corrupt state quietly are a different class of problem from the ones that error loudly, and why "did you test the sad path" isn't enough if you only tested the sad paths that produce visible errors.

 The artifact distinction is the real one. Everything FetchSandbox does assumes the event log exists, webhook delivery records, response payloads, retry sequences. That's exactly why replay and receipts work at the integration layer: there's something to replay from.

The call/notification class breaks that assumption entirely. No artifact means the exposure window isn't just hard to measure, it's structurally unobservable after the fact. You're right that the only real answer there is capturing as it happens, not reconstruction. That's a different category of problem, and trying to retrofit the diff-and-count approach onto it gives false confidence.

Where I've landed for FetchSandbox: scope it to the layer where logs are guaranteed and don't pretend the receipt proves anything about interactions that left no trace. Curious whether you've seen teams actually instrument the call layer in a way that makes post-hoc scoping tractable, or if it's mostly just "hope someone filed a ticket."

@Raj Nagulapalle that's the real distinction - a loud error gets triaged same day, a silent state change just sits there compounding until someone notices the number looks wrong. makes me wonder if there's a category of tests worth writing specifically for "did this action leave a trace anywhere", separate from whether it produced the right output. an audit log check as its own test, basically.

 yeah .. "did this leave a trace" as its own assertion, separate from "was the response right," is exactly the right cut for the silent class. it's what the chaos invariants already do under the hood... they don't assert on the API response, they assert on the resulting state ,.. "is this sub still canceled after the stale event," not "did the endpoint return 200." but your framing is cleaner. most suites assert on outputs because outputs are what the code hands back; the silent-corruption bugs live in the side effects nothing returns. a first-class state/trace assertion .. did the action leave the record it should, and not leave the one it shouldn't — catches a category output-assertions structurally can't. might steal this framing.

@rnagulapalle the exposure-window framing is the right shift, but it assumes you can measure exposure after the fact - for structured state that's true, you can diff timestamps and count affected rows. for something like a call there's often no artifact at all once it's over, no record of what got said badly, just a support ticket if the person happens to complain. so for that class the problem isn't catching faster, it's that you can't even scope "how much got out" unless you were capturing the interaction as it happened, not reconstructing it after.

 yeah that's exactly the wall i keep running into when people ask about live calls. everything i do assumes i can re-fire the exact webhook ordering because it was captured as typed events in the first place. diff the timestamps, count the affected rows, there's an artifact. a live call has none of that once it's over. you can't scope the blast radius unless you captured the interaction as it happened, and reconstruction is just guessing.

so the whole reproduce-and-prove model only works if the interaction was capturable as replayable state to begin with. where it isn't, the leverage moves upstream to live capture. genuinely different problem. feels like our two use cases meet right at that line

#2, the out-of-order one. we had a subscription webhook handler that assumed events arrived in the order they were created, because in dev/staging they basically always did. in prod under load they didn't, and for about two weeks we had users who paid getting a "not entitled" screen until a retry or a manual refresh happened to land after the right event showed up. tests all passed because the test suite fired events in the "sensible" order too - nobody wrote a test for the handler receiving them scrambled. the fix that actually stuck wasn't more tests, it was making the handler idempotent and order-independent by keying off the subscription's current state rather than trusting event sequence at all

 two weeks of paying users hitting "not entitled" is brutal. and the fix you landed on is the right one, keying off the subscription's current state instead of trusting event sequence is how this class of bug stops coming back. the ordering is a symptom; the real problem is handlers that have an implicit belief about what "just happened" based on which event arrived. that's exactly the scenario the permutation fuzzer in FetchSandbox is designed to surface before you ever ship it, fire every ordering, diff the final state, see which ones land somewhere unexpected.

 fire every ordering and diff the final state - that's the part I like, because it doesn't require you to predict which ordering is dangerous ahead of time, it just surfaces whichever ones diverge. curious how deep the permutation goes though - full event history is often way more than 2-3 events, so are you capping how many you shuffle or is there some heuristic for which subsets are worth permuting instead of brute forcing all n!

 not the full n! .. that'd blow up exactly like you're picturing. the trick is we don't permute the whole event stream, only the events that mutate the same resource's state... and those are the only ones whose ordering can change the final entitlement .. events touching different resources don't need cross-shuffling. so it's permute-within-the-interacting-set → prune orderings that can't diverge → cap the window if a resource somehow has a pathological number of events → minimize to the 2–3 that reproduce. "permute what can actually interact," not brute-force the universe. if you've hit a case where the interacting set itself is huge, I'd genuinely want to see it .. that's where the heuristic has to earn its keep

@rnagulapalle glad it was useful. it maps to something I keep running into with real-world side effects too - a state assertion is checkable after the fact, but there's a category above that where even the corrected final state doesn't undo what already happened out in the world. the record can say entitlement corrected, but that doesn't erase the two days someone had free access, or in my world, a phone call someone already heard. the test catches the bug, it doesn't undo the blast radius that already happened before it was caught.

 the 'already-in-the-world' class is where the blast radius calculus breaks from state-correctable bugs entirely. correcting entitlement doesn't un-give two days of free access. the call your user already heard doesn't un-ring. what changes the contract isn't just catching faster, it's knowing the earliest possible detection point so you can scope real exposure, not 'did we catch it' but 'how much got out before we did.' that's the thing most test receipts don't surface, and probably should.