Agent-written webhook handlers keep breaking in prod. How are you testing them?
Spent way too long this week watching devs debug Paddle billing issues that never reproduced locally.
The pattern keeps repeating: subscription.activated arriving before subscription.created in prod, about 30% of the time. Handlers look clean, pass unit tests, handle retries correctly. But they assume events arrive in order. Paddle's sandbox just doesn't reproduce that chaos.
This comes up constantly with teams building on Lovable, Bolt, Base44.
The AI writes the webhook handler, it looks solid, gets shipped. Then prod explodes and nobody can reproduce it because the local environment is too clean.
We built FetchSandbox to inject exactly this kind of chaos against a stateful Paddle twin. Out-of-order delivery, retry storms, concurrent events hitting the same subscription. Two race conditions caught before they hit customers in the first week.
Curious what others are doing though. Are you testing event ordering explicitly? Replaying prod events? Or mostly monitoring and accepting some prod chaos as the cost of shipping fast?
How are you validating that agent-written webhook integrations actually survive in prod?
Replies
Hey. Testing the ordering is the part I'd push back on, 'cauz ordering isn't something either end can promise you. Stripe says outright that it doesn't guarantee events arrive in the order they were generated, and your 30% is Paddle telling you the same thing without writing it down. A handler that survives your chaos run is still one retry storm away from a sequence you didn't think to inject.
For me was making the order stop mattering. The handler doesn't apply what the event says. It takes the object id out of the event, fetches the current state from the API, and writes a row that's a function of that fetch. subscription.activated landing before subscription.created then does the same thing either way, because both events ask the same question and get the same answer.
The failure that survives that is the one I'd point your chaos at instead. Two events for the same subscription, both fetch, and the older read writes last. Nothing arrived out of order, nothing retried, and the row is still wrong. Guarding the write on the fetched object's own updated_at, so a stale read can't overwrite a fresher one, is what fixed it for me.
@siarheihamanovich Yeah that's actually how our probe engine works already, every run fetches current state from the provider before injecting anything so the twin always has real data to reason against. So we're aligned there. The concurrency one though, two concurrent fetches where the stale write wins, that's something we keep hearing from the dev community too. Nothing looks wrong until the row is just quietly wrong. Adding that guard to the suite, appreciate you spelling out the updated_at check specifically.
The agent-written part is doing more work here than the webhook part. Every doc sample and every tutorial applies the event payload directly, so that's what a model writes by default, and Siarhei's fetch-the-object pattern basically never comes out unless you ask for it by name. I'd rather catch that at review than in a chaos run, and the cheap version is a rule that fails any handler reading fields off the event body other than the id. Then the chaos suite only has to cover the concurrent stale write, which is the one you can't lint for.
@asadmalik901 good split. the rule scope probably matters though: blocking all payload field reads is too broad since some integrations pass fields you genuinely don't need to re-fetch. the narrower flag is event data used as authoritative state. for the concurrent case, the test that actually catches it stages two reads so the older result commits last, then checks the newer state survived.