We ran a Lovable checkout app through FetchSandbox. Six checks passed. Two stayed not checked.
We delayed an email response, but the webhook still returned success. We hadn't shown it recovering after a failed callback. So the overall run stayed incomplete.
That's the thing I want us to get right as we build this. Did the test cause the failure, or did the app just succeed? Same question for a background job that might crash halfway through.
What evidence do you ask for before trusting a recovery test?
thanks man!!!, genuinely thoughtful review, and the WireMock/Postman/Merge comparison nails why this exists, mocks hand back a 200 and cant touch stateful race conditions.
honestly your needs improvement section is right, so let me just agree in the open:
1/custom/proprietary specs, correct. you can import any openapi spec and the basic calls run, but the deep curated failure library (the real dup/late/stale reproductions) only exists for the seeded set today. widening that, and getting to generic proof for any imported spec, is the work ahead.
2/autonomous capture, also correct. today its curated and proof-gated, you point it at a known failure mode and it reproduces + proves. learning arbitrary failures from your own runs in the background is the direction, not shipped, and i wont pretend it is.
3/one you were a little generous on, ill call it myself, deterministic out-of-order replay isnt live yet. it falls out of timing today, not a pinned deliver-this-before-that sequence. its literally the next thing im building.
4/and the concurrency/observability point is fair, the terminal view gets thin under heavy parallel runs, better telemetry there is on the list.
appreciate you taking it seriously enough to find the real edges. thats worth more to me than the stars.