Agent-written webhook handlers keep breaking in prod. How are you testing them?
Spent way too long this week watching devs debug Paddle billing issues that never reproduced locally.
The pattern keeps repeating: subscription.activated arriving before subscription.created in prod, about 30% of the time. Handlers look clean, pass unit tests, handle retries correctly. But they assume events arrive in order. Paddle's sandbox just doesn't reproduce that chaos.
This comes up constantly with teams building on Lovable, Bolt, Base44.
The AI writes the webhook handler, it looks solid, gets shipped. Then prod explodes and nobody can reproduce it because the local environment is too clean.
We built FetchSandbox to inject exactly this kind of chaos against a stateful Paddle twin. Out-of-order delivery, retry storms, concurrent events hitting the same subscription. Two race conditions caught before they hit customers in the first week.
Curious what others are doing though. Are you testing event ordering explicitly? Replaying prod events? Or mostly monitoring and accepting some prod chaos as the cost of shipping fast?
How are you validating that agent-written webhook integrations actually survive in prod?
Replies
Hey. Testing the ordering is the part I'd push back on, 'cauz ordering isn't something either end can promise you. Stripe says outright that it doesn't guarantee events arrive in the order they were generated, and your 30% is Paddle telling you the same thing without writing it down. A handler that survives your chaos run is still one retry storm away from a sequence you didn't think to inject.
For me was making the order stop mattering. The handler doesn't apply what the event says. It takes the object id out of the event, fetches the current state from the API, and writes a row that's a function of that fetch. subscription.activated landing before subscription.created then does the same thing either way, because both events ask the same question and get the same answer.
The failure that survives that is the one I'd point your chaos at instead. Two events for the same subscription, both fetch, and the older read writes last. Nothing arrived out of order, nothing retried, and the row is still wrong. Guarding the write on the fetched object's own updated_at, so a stale read can't overwrite a fresher one, is what fixed it for me.
We started requiring chaos tests in PR review. Would you be open to sharing how you model the Paddle twin state so others can replicate that out of order logic?