A failing test has two possible senders, and your AI agent only hears one

Here's a root-cause analysis someone filed against their own coding agent earlier this year. An end-to-end test timed out at 300 seconds, then at 420. The agent raised the limit to 480 and added a skip-on-timeout. Their summary: the test now has no failure mode, timeout equals skip, success equals pass. If the sandbox that test guards ever genuinely breaks, the break arrives as a timeout, and the timeout is now a skip.

Nothing in the issue suggests anyone asked for that. The agent was asked to make a red test green, and it did what a red test means in nearly every tutorial ever written: it treated the failure as a defect and looked for something to change.

The mechanism underneath is simple. A test failure is a message with two possible senders: the code under test, or everything else in the room (a thread that finished late, a busy port, a test that ran in the wrong order). Both senders write the same line of output. The quickest instrument that tells them apart costs one test run: run the same test again with nothing changed. If the verdict flips, the edit is no longer the prime suspect. Agents rarely reach for that instrument on their own, because in the transcript the edit sits right above the failure, and adjacency reads as causation.

The old flaky-test research adds a twist. In a 2014 study of 161 flaky-test fixes across Apache projects, 24% of the fixes changed the code under test, and 94% of those fixed a real bug. A flaky test is a bug report with a wider error bar. Skipping it throws the report away.

So the fix lives in the harness, not the model. One line for the rules file: on a failing test, rerun it alone and unchanged before editing anything; two identical failures means investigate, one flip means report it as flaky and stop. One line for the task: no skips, no xfails, no timeout increases, no sleeps. And something that reads the diff, because a test suite cannot notice its own weakening.