The tests my AI wrote were passing on broken code
I'm not a developer. I build by directing AI agents, so tests are how I know something works. I can't check it by reading the code.
For a while the green checkmarks were enough for me. They shouldn't have been.
Now every bug fix comes with a test. Before I accept it, the bug gets put back on purpose and the test has to fail. When we started doing that, we found tests that had been passing on broken code the whole time.
Someone on Indie Hackers put it better than I could:
> a green suite the agent wrote is agreement with itself
Another one that paid off, also suggested there: log in as one user and try to open another user's data by URL. That found a real bug. It's fixed now, with a test watching it.
What's the one check on AI-written code that caught the most for you?
Replies
An agent made a failing test pass by changing an exact match assertion to check if the field just existed. The suite stayed green for days while the calculation was broken. Git diffs on test files have to be reviewed before looking at the implementation.
The check that caught the most for me was moving the test off the code and onto a device: an end to end run on a real Android phone against the real backend, a scenario that signs in, makes a song and plays it. Two things it showed that unit tests had blessed. A credits field set by hand showed 100 in the app while the backend still answered not enough credits, because the backend counts a ledger and the screen reads a number. And the test harness itself could swallow errors: a build exception during the walk made the run report an unrelated assertion and then hang, instead of failing. A test that cannot fail properly is the quiet cousin of the one that passes on broken code. Your other user by URL check is the one I would keep forever. My side of it is deny everything by default in the database rules and let only the backend write credits, after an earlier app of mine let the client write its own purchase counters and paid for it.