Our prioritization system passed every spec check and was completely useless

We built a feature that sorts business email into five levels of urgency — act now, review today, monitor, routine, no action. Before turning it on for anyone, we ran it across 6,037 real emails to see how it behaved.

86.1% landed in a single tier.

Every rule fired correctly. It did exactly what the specification said. And it would have been worthless, because a system telling you that 86% of your inbox is equally important has told you nothing at all.

Two things I took from it.

The spec was the problem, not the code. We had written rules for what should be urgent instead of measuring what people actually treat as urgent. Recalibrating against real inbox behavior is what finally spread the distribution out.

We were also scoring the wrong thing. Messages that had already been replied to or archived were still counted as demanding attention. We were rating the email, not the situation. Fixing that mattered as much as the recalibration did.

The uncomfortable part: we only caught this because we looked at output distribution. Our accuracy metrics looked fine the entire time, and would have kept looking fine after launch.

So a question for anyone else shipping classification or ranking inside a product — what do you check before you trust the output? Is distribution analysis a standard step in your QA, or did we just get lucky catching this pre-launch?