A model router should optimise for boring, not cheap
Every multi model product ends up building a router, and the first version always optimises for cost. Cheapest model that can plausibly handle the request. It works, and then it doesn't, in a way that's hard to see. The answer comes back confident and slightly wrong, nobody complains because nothing looked broken, and they quietly stop coming back.
What I actually want from a router is consistency. Same question, same shape of answer, tomorrow as well as today. Someone who has learned what your product is bad at can work around it. Someone who gets a different quality of answer every time can't learn anything, and that's worse than being reliably mediocre.
So the rule I've landed on is that a route only changes when I can explain why to the person using it, and cost isn't a reason I can say out loud.
The open question for me is detecting the plausible but wrong case without a human reading outputs. Everything I've tried is either a second model grading the first, which shares the blind spot, or a rule that only fires on mistakes nobody was going to make. If you're routing across models and have solved this, I'd like to hear how.
Replies
I've always felt that saving a few cents isn't worth much if the user gets a worse answer. for me the router should first figure out what each model is reliably good at then use cost to fine-tune the decision.
@jessicaflou0 Agreed on the ordering, but the trap is that reliably good at isn't a fixed property of a model, it's per task and it drifts. We stopped keeping a global ranking and kept a small scorecard per route instead, because the same model was our best at one job and our worst at another. Cost as a tiebreaker between two routes that already pass is fine. Cost as the first filter is exactly how you end up shipping the confident wrong answer.