The cheapest model on your router is usually the most expensive one you run
Everyone read today's pricing news as costs going down. On agent workloads it doesn't work like that.
A cheap model that gets something wrong rarely fails loudly. It returns something plausible, the next step eats it, and you find out four steps later. By then you've paid for the cheap call, the three calls after it, and the rerun on the expensive model. A call that errors out is nearly free by comparison, because you just retry it.
So the routing question isn't which model is cheapest per token. It's which model I trust enough to not verify. Anything below that line needs a checker, and the checker is usually a bigger model, which is exactly where the savings go.
Where I've landed: cheap models for anything I can validate mechanically, like extraction where the schema either parses or it doesn't. Expensive models for judgement, where the only validator is a person reading it a week later and going "hang on".
The tier I still can't make work is the middle, cheap model plus cheap checker. Every time I try it, the checker agrees with the thing it should have caught.
Replies
I’ve had the same experience. The better routing rule is probably not cheap vs expensive, but verifiable vs judgment-heavy. Small models work well when you have hard checks like schemas, tests, or exact matches. But once an output influences another agent or reaches a user, I’d escalate early. Tracking retries and repair calls per model also helps — the “cheap” route often stops looking cheap once those are included.
@alpertayfurr verifiable vs judgment-heavy is a better line than mine, I'm stealing it. Once we logged repair calls as their own line item the cheap tier came out around 1.7x its sticker price on anything freeform, and that still doesn't count the twenty minutes I spent reading output that made it through. The boundary I now escalate at is the same one you named, the moment output leaves the machine and hits a person.
@asadmalik901 That 1.7x number is exactly why sticker price can be misleading. I’d probably track cost per accepted output instead: model spend, retries, repair calls, and human review time combined. Once output reaches a person, even a small quality gap becomes expensive very quickly.
@alpertayfurr
@alpertayfurr Ignore the blank reply above, phone misfire. Cost per accepted output is the right unit, the trouble is defining accepted. Most of our runs have no clean accept signal, nobody clicks anything, they just quietly re-run the prompt with two words changed. The proxy we landed on is whether the output got edited within ten minutes, which is crude but it flipped the ranking of the two models we were comparing.
"which model I trust enough to not verify" reframes it well. The routing decision I keep getting wrong is judgement calls, where I've been trusting a cheap model just because the output reads confidently.
@irahimiam Confidence is the tell. The one that cost me was a summariser that never once said it didn't know, it just filled the gap with something plausible and carried on. I now weight models partly on how often they refuse, not just how good the good answers read.