Three monitors I trusted were incapable of reporting failure

I spent a week fixing three separate monitoring bugs and they turned out to be the same bug wearing different clothes. In all three the instrument could only ever return good news, and none of them looked broken.

One. A refresh job reported one item refreshed successfully. I checked the field that records when the price was last written and it had not moved. The function underneath returns an ok status for a product that does not exist, and my counter incremented on ok rather than on a value having been written. So the counter measured whether the call returned, not whether anything happened. It had been reporting success for as long as it had existed.

Two. A scraper node was supposed to rotate its address after being blocked too often. It had rotated zero times in twelve hours. The block counter was fed in the general error path, but a captcha was caught by an earlier branch that returned before reaching it. So the one failure the entire mechanism existed for was the one failure it could not see. The node called itself healthy the whole time it was being turned away.

Three. My public status page showed a degraded service for weeks. The service was fine. Worker identity was keyed on hostname, and inside a container the hostname is the container id, so every restart registered a new machine while the old rows stayed forever, ageing and unhealthy. The status page was correctly reporting on machines that had not existed for weeks.

The shape is the same in all three. Each instrument watched something adjacent to the thing I cared about. Call returned instead of value written. Generic error instead of the specific refusal. Row present instead of machine alive. The adjacency is invisible because the two agree almost all of the time. They diverge exactly when something is wrong, which is the only moment you are actually reading the thing.

The check that has caught all three since is one question: what would a total failure of the underlying thing look like on this dashboard? If the answer is a smaller number rather than an obviously different reading, the instrument is counting attempts. If I cannot describe the reading that failure produces, the instrument cannot produce it.

The only structural fix I have found that generalises is to derive the metric from the artifact instead of the code path. Read the timestamp on the record, not the return value of the function that was supposed to write it. One of those can lie to you and the other cannot.

Curious whether anyone has something better than case by case paranoia here.