I wrote a script to find checks that can't fail. The script couldn't fail

Someone on another thread described a test he uses. Revert the fix, leave the test in place, and see whether the test still passes. If it does, the test was asserting something unrelated to the bug it claimed to cover.

Good idea, so I pointed it at all eleven gates in my own pipeline. Introduce exactly the defect each one claims to catch, confirm that specific check goes red, revert.

First run: four caught, three apparently broken. Second run, after I aimed the mutations properly: all three still green. I was about a paragraph into writing up which of my own checks were dead. Then I looked at the script.

I was filtering its output with grep -iE 'FAIL' piped into grep -vE '^\s+'. Every line my gate prints starts with two spaces. So the second filter was discarding exactly the lines the first one had just found. The variable holding the failures could never be anything but empty.

I had written a detector that could not report a failure, in the middle of an exercise about finding things that cannot report failures. Third attempt, with one change at the top: before trusting any result, run the detector against a state I already know is broken, and abort if it comes back green. That step found nothing on the third run because by then the script was correct. It would have caught the second run instantly.

All eleven gates hold, for the record. The only dead check found that evening was the one I wrote to find dead checks.
The rule I'm keeping: breaking your check on purpose isn't enough. Break the thing that reads your check as well. Anything that reports on a system is part of the system.