Everything that broke today had already reported success

Writing code got cheap. Checking whether it did the thing did not, and the gap between those two is where I spent a whole day.

None of what follows was broken. Every line ran as written, raised no error, and reported success. That is the part worth talking about, because I think it is the real failure mode of this era and it does not look like a bug.

One. A refresh job reported one item refreshed. I checked the field that records when a price was last written and it had not moved. The counter incremented when the call returned rather than when a value was written, so it had been reporting success for as long as it had existed. It could not have reported anything else.

Two. A refresh path read one field name while the scraper was sending a different one. One word apart. When the price failed to parse, the missing field fell back to a default meaning the product does not exist, so live and in-stock listings were stamped as dead and pulled off a storefront. There was a comment directly above it explaining that a three strike threshold protects against exactly this. The threshold does nothing, because the validity check in the same function also requires a price and there was none. One event was enough. I found it by triggering it on my own site and taking two working products down.

Three, and this one humbled me. I wrote a script to verify case two and measure how many affected links were still alive. It reported two out of nine. I read the raw output and six of the nine had come back with a real product title, which already proves the page exists. My script only counted a link as alive if a price came back. I made the same mistake I was in the middle of diagnosing, in the tool I built to diagnose it, hours after writing a long post about that exact pattern.

What caught all three was not more testing. Repeating a call cannot find any of them, because a deterministic wrong answer passes every repeat perfectly, by construction. Repetition detects instability and is blind to mislabelling.

What worked was a control. A second arm, run at the same time, that should come back different. A real identifier next to an invented one in the same batch. The field the code writes next to the field that records the outcome. The count my script produced next to the raw rows it produced it from. In all three cases the truth was already in the room and nothing was comparing it to anything.

I do not think this is about tooling, or about who or what wrote the code. It is that we have become very good at producing confident output and have not improved nearly as much at asking output to disagree with something.

So what is your control? Not the test suite. The thing you put next to a result to find out whether the result means what it says.