We got our tool to write its own bug-reproduction tests
The gap most auto-fix tools have
They tell you the bug is fixed. They don't show you. Our rule has always been: make the bug happen on real code, apply the fix, show it stops happening. The gap was we could only do that for bugs we'd scripted in advance. Anything unusual and the honest answer was "found it, fixed it, can't prove this one."
What we built
We taught FetchSandbox to write the reproduction itself, no pre-scripted test required.
The safety check that makes it trustworthy
A generated test could be wrong. It might pass broken code and hand you a fake green. So before we trust it, the test has to actually catch the bug on the broken code first. If it can't, we throw it away.
Real example
Ran it end-to-end on a billing bug: negative seat count slipping through and shrinking a customer's plan. It found the code path, drove the real handler, stubbed only the db as a passive recorder so the app's own logic decided the verdict. Confirmed the reproduction before trusting it. No hand-holding.
Replies
I like that the generated test has to fail befor it's trusted. That extra check makes the whole workflow feel much more reliable.
FetchSandbox
@ill_robyn Thanks man... that one rule does most of the heavy lifting. A test that can't fail on broken code can't be trusted to pass on fixed code, so everything else hangs off it. Glad that's the part that landed.
How does FetchSandbox handle bugs that depend on timing or race conditions? Those are usually the hardest ones stop reproduce.
FetchSandbox
@christian_onochie thank man n goood one, and it splits. A lot of what looks like a race in an integration is really an ordering bug a webhook arriving twice, out of order, or before the thing it references. Those are deterministic once you stop trying to hit the window and just fire the events in the pathological order; the probe controls the sequence instead of racing it. That covers most of them and we catch those reliably. Genuine nondeterministic concurrency (shared state, thread interleaving) is the harder tail .. if the bad interleaving can be forced/serialized we can reproduce it, but I won't pretend we've solved true timing-dependent races. Those still need controlled fault injection, and some still need a human.
Dial
the throw-it-away-if-it-doesn't-fail-first check is the right instinct, that's the part that keeps a generated test from lying to you. curious about a different kind of hard case though: bugs that only show up with a specific shape of production data rather than a specific timing window, something like a currency field that's a string in 90% of rows and a number in the rest. does the generator ever get to see anonymized samples of real data distributions, or is it working purely from the code path, in which case would it even know that shape exists to reproduce it?
FetchSandbox
@omri_ben_shoham1 thanks again.. this is closer to the edge than timing, honestly. ..today the generator works from the code path + known failure classes, not your real data distributions .. it doesn't peek at samples. What it can do is reason about what the code assumes vs. what it guards: if a handler does float(row['amount']) with no type check, "amount is sometimes a string / sometimes null" is exactly the shape it'll try, because the fragility is visible in the code even when the distribution isn't..type-coercion is a known class, so that one it'd likely catch. What it can't do is know a shape that only lives in your prod data and isn't betrayed by the code — your 90/10 string-vs-number split, if nothing in the code hints at it, it wouldn't know to try....feeding it anonymized schema/value stats would close a lot of that, and we don't do it ye..genuinely on the list ... good prompt.
The fail-first check is the part that actually makes this trustworthy, not just cool — a generated test that can pass on broken code is worse than no test.
Two things I'm curious about. First, once a repro test is confirmed against the real bug, does it get promoted into the permanent suite as a regression guard, or is it scoped to that one fix-and-verify run and then thrown away? Second, is the reproduce-then-verify step something only FetchSandbox's own fix flow can call, or could an external agent hit it directly to check its own patch before claiming a fix works? Would be a genuinely useful trust primitive to have exposed on its own.
FetchSandbox
@akbar_b Both are exactly how I think about it.
Right now a confirmed repro is scoped to that fix-and-verify run — the receipt is the durable, re-runnable artifact, but we don't yet auto-promote it into a standing regression suite. That promotion (a proven repro becomes a permanent guard) is the obvious next step, and it's the whole "remember what breaks" thread.
Yes — the reproduce-then-verify step is a standalone MCP tool (prove_fix), so an external agent can hit it directly to gate its own patch before claiming a fix works, not just our own fix flow. It's on npm. That "trust primitive exposed on its own" framing is exactly the bet — glad it read as the interesting part, because it's the part I care most about
@rnagulapalle Love that prove_fix is standalone on npm — that's the piece I'd actually reach for. On the promotion step you mentioned: the fail-first check is what makes a repro trustworthy, but it only runs once, at birth. After it's promoted to a standing guard and the code drifts around it, it could start passing because it's quietly stopped hitting the bug — the same fake-green you designed fail-first to catch, just showing up later in the test's life. Do you picture a promoted guard periodically re-proving itself against a pinned broken snapshot, or is that a separate problem from where you're focused right now?
The fail-first check is what I'd actually want pointed at our own codebase. We've got a bunch of the largest service files with zero dedicated tests, everything else that touches them just mocks them out, so a green suite tells me nothing about whether the real logic in those files ever ran. A tool that has to prove its test fails on the current code before it's trusted would catch a lot more than what we have now, which is basically hoping the mock matches reality.