roughly half the debug loop has been missing from coding agents
i build dev tools solo and use claude/cursor daily for API integration work. the pattern i keep seeing: agent finds a bug, fixes the code, declares done. what it does not do is rerun the exact failure scenario and prove the fix holds. that verification step was always on me.
we just shipped something in FetchSandbox to close that loop. after it catches a bug, it now nudges the agent to rerun the same request and surface a behavioral comparison, before vs after. same scenario, does it pass clean? that receipt is now automatic.
my rule: finding the bug is table stakes. proof that the behavior changed is the job. "i fixed the code" and "the fix works" are not the same claim.
curious how others handle this. do your agents ever rerun failure scenarios to verify, or does that step still fall back to you?
Replies
in my experience the agent almost never reruns the original failure on its own, it moves on to the next thing the second the fix compiles or the test suite goes green. the gap is exactly what you're describing, a green test suite proves the new code works in isolation, not that the specific broken request from 10 minutes ago now behaves differently. the before vs after receipt idea is the right fix because it forces the comparison to be the actual failing case, not a stand in test someone wrote after the fact.
+1 to Omri's point that the agent moves on the second it goes green — and even once you force the rerun, "the same scenario" is trickier than it sounds: a lot of bugs only fire from a specific starting state — a stale cache row, a half-applied write, an account already over its limit — and both the failing run and the fix tend to change that state. So replaying the same request against the now-current world quietly takes a different branch and passes, without ever recreating the condition that broke it.
That makes "before vs after" risky: if the two runs don't start from the same world, a green can mean "fixed" or just "the precondition that triggered it is gone." The receipt I'd actually trust pins the starting state, not only the request — same input and the same world it failed from. Does the replay snapshot the state the scenario began in, or just re-issue the call? Really nice work either way 🙏
FetchSandbox
omri's point is the mechanical reason the loop breaks. akbar's is the harder one: "same scenario" isn't same-world. for the scenarios FetchSandbox defines today (stripe duplicate webhook, idempotency failures), the stale starting state is baked into the scenario spec itself, so replaying hits the same branch rather than the now-current world. that only holds because we control the precondition definition. if the broken state lives in the user's own DB rather than the API interaction layer, we're not freezing that yet. the real_app proof mode we just shipped captures before/after of the user's actual code behavior, but pinning arbitrary starting-state snapshots is the next honest gap to close.