The cheap model is wrong in a way the expensive one isn't, and it won't tell you

We route across a handful of models and on a spreadsheet the savings look obvious. The small model gets it right most of the time. The problem isn't the hit rate, it's the shape of the misses.

When a frontier model is unsure you can usually see it. The output hedges, goes vague, asks a question. When a small model is unsure it produces something clean, confident and wrong, and that sails straight past any eval checking format instead of truth. Nobody catches it until a user does.

So the real cost of routing isn't the token delta. It's that the cheap path now needs a verification step, and verification is another call, which eats the margin you routed for in the first place.

What's worked for us: route on how checkable the task is, not how hard it looks. Anything with a deterministic check, parseable output, a schema, a test that passes or fails, goes cheap. Anything where being wrong stays invisible until someone complains goes to the expensive model every time, no exceptions, even when it looks trivial.

Has anyone found a routing signal that beats checkability? Difficulty scoring never worked for us.