My Agent Wrote a Test That Passed. I Reverted the Fix. It Still Passed.

I was reviewing an agent's work on a bug fix. Clean diff, passing test, looked fine.

Something made me suspicious. So I reverted the fix and ran the test again. Green. The assertion was checking the input parameters, not the result. The bug was still in production.

Now I have one rule: if the test doesn't go red without the fix, it isn't testing the fix. The agent is good at writing code that looks like a test. It's not always good at writing a test that would catch the actual bug.

How many green checkmarks have you trusted that were actually decoration?