When does using a bigger LLM stop making sense?
I’ve been wondering how people decide when to switch to a smaller model instead of just using the best one available.
A bigger model might handle edge cases better, but it also costs more and takes longer to respond. For simple tasks, that trade off doesn’t always seem worth it.
How are you guys approaching this?
12 views
Replies
I decide per call, not per product. My backend has about ten model calls and almost all of them run on a Flash model: writing lyrics, a title, describing a photo, rescuing a prompt the music engine rejected. One tiny step runs on Flash Lite. Only one call uses the Pro model, the one that aligns lyrics to the audio for timed lyric videos, because a line that is half a second late is visible to everyone and there is no human to catch it. So my rule: the big model goes where a wrong answer is both likely and expensive to notice. Everywhere else the smaller one wins, and the latency matters more than people expect.
Ideal way is to have a mixed architecture, like based on the complexity of the task,
But in a low latency environments - like in voice ai calls, i am a bit confused if we can have a decider based on the complication or type of task and use the appropriate llm from the list of available ones, is this designable?