The cheap model in your router is the most expensive one you run
We started routing the easy stuff to a small model to save money. Cost per request dropped about 40% and the dashboard looked great. What the dashboard didn't show was people re-sending the same message, so once you count the retry and the follow up clarification, the cheap path burned more tokens than just sending it to the big model once.
The mistake wasn't the routing. It was measuring cost per request instead of cost per finished task. A request that has to be asked twice isn't cheap, it's a request plus a support ticket forming.
We still route. We just stopped routing on what the prompt looks like and started routing on whether the last cheap answer got a follow up within 30 seconds. That signal is dumb and it beats everything we tried to infer from the text itself.
If you're routing, what's your retry rate on the cheap path? I've never seen anyone publish theirs, which makes me think the number is embarrassing for most of us. Mine was 31% before we changed it.
Replies
No router here, and the reason might be more useful to you than a retry rate would be.
Your 30 second follow up signal works because a bad chat answer announces itself. The user reads it, sees it's wrong, and re-sends. The signal exists because the failure is visible to the person who received it.
Mine isn't. The output is a written case study built from a customer interview, and the expensive failure is a version that reads perfectly well and quietly overstates what the customer actually said. Nobody re-sends that. They publish it. The retry rate on that failure is zero, and zero is exactly the wrong number to be optimising against, because the damage happens downstream in someone else's inbox weeks later.
So we don't route, but the thing I'd actually take from your post is the reframing rather than the tactic. Cost per finished task is right, and the harder question is whether your product even generates a signal when the task wasn't finished properly. If it doesn't, a cheap path can look clean forever.
Worth asking anyone who quotes you a low retry rate: is that because the cheap model is good, or because your users can't tell?
@venkat_a2 Is the model good or can the user not tell is the question I should have opened the post with. Chat gets away with a dumb signal because the reader is the reviewer and they review instantly. Anything that produces a document for a third party has no reviewer in the loop at all, so you have to manufacture the signal rather than observe it. The cheap version for you would be a second pass that only checks whether every number in the draft appears in the transcript, then track how often it disagrees with the writer. That disagreement rate is your retry rate, and it's probably the only one you'll ever get.