The cheap model is wrong in a way the expensive one isn't, and it won't tell you
We route across a handful of models and on a spreadsheet the savings look obvious. The small model gets it right most of the time. The problem isn't the hit rate, it's the shape of the misses.
When a frontier model is unsure you can usually see it. The output hedges, goes vague, asks a question. When a small model is unsure it produces something clean, confident and wrong, and that sails straight past any eval checking format instead of truth. Nobody catches it until a user does.
So the real cost of routing isn't the token delta. It's that the cheap path now needs a verification step, and verification is another call, which eats the margin you routed for in the first place.
What's worked for us: route on how checkable the task is, not how hard it looks. Anything with a deterministic check, parseable output, a schema, a test that passes or fails, goes cheap. Anything where being wrong stays invisible until someone complains goes to the expensive model every time, no exceptions, even when it looks trivial.
Has anyone found a routing signal that beats checkability? Difficulty scoring never worked for us.
Replies
A clean paragraph that's factually off is the failure mode I dread most in copy review too, it slides past a skim way easier than something that visibly hedges. I've started grading AI drafts by how confident they sound, not just by their typos.
@yelyzaveta_kibets Grading on confidence works until you tell the model to hedge, because then it hedges uniformly and you've lost the signal instead of fixing it. What held up better for us is asking the draft which single claim in it is weakest, since a model that's fine says something trivial and one that's bluffing usually names the invented bit. Not reliable enough to skip a human, but it's cheaper than a second full pass.
This is a great point. The real issue isn’t how often the cheaper model is right, but how easily you can catch when it’s wrong. Checkability seems like a much more practical routing signal than difficulty
@aruna_polosofttech The catch I didn't expect is that checkability quietly reshapes what you build. Anything you can't verify cheaply starts feeling expensive, so you stop proposing it, and I've caught myself picking the schema shaped version of a feature for routing reasons rather than product ones. Still the right default, just worth knowing you're making that trade.