Do we need an LLM for every decision in an agent?
by•
I’ve been thinking about this with agent harnesses. A lot of the work between LLM calls is just deciding what happens next: retry, switch tools, pick another model, or stop.
Some of these decisions seem simple enough for a small model or even normal code. But getting one wrong can mess up the entire run.
Would it make sense to use a small, fast model for these Type 1 decisions and save the bigger reasoning model for the hard stuff?
Where would you draw the line? I’d be interested in how people are handling this in their own setups.
12 views
Replies
I'd separate known states from ambiguous ones. A missing required field can go straight back to the user in code. A small model could classify an unfamiliar failure, but let it return unknown rather than force a retry or tool switch.
I think a small model for simple decisions is the best balance between cost and reliability
My pipeline splits it in three, and the split follows what the decision needs to know. Code decides everything that is about mechanics: a dropped connection on a read is retried up to three times, and a request that may already have reached the provider is never repeated, because that could bill twice. Code also recognises the provider's refusal messages by their wording. Only then a small fast model gets the one judgement code cannot make: is this refused prompt genuinely prohibited, fine as it is, or worth rewriting once. Its answer can only pick from those three, and "prohibited" ends the run with no retry. The big model never sits in that loop. The rule I use: a decision goes to a model only when it is about meaning, never when it is about state.