Multi-LLM routing in practice, Claude for reasoning, Llama for cost, what the routing logic like
by•
Skipping the marketing version, here's how it actually works in our production deployments.
Routing logic:
- Task classification at agent input: reasoning-heavy, retrieval-heavy, generation-heavy, or sensitive
- Reasoning-heavy goes to Claude 3.5 Sonnet
- High-volume retrieval-heavy goes to Llama 3 70B (cheaper at scale)
- Sensitive (PII, regulated) goes to local Llama for on-prem customers
- Generation-heavy with brand voice goes to GPT-4o
The wins:
- 35 to 60% LLM cost reduction across our customer base
- Better latency on high-volume tasks (Llama serves faster locally)
- Compliance posture improves without sacrificing capability
The tax: more testing surface area. Worth it, but the testing cost is real.
4 views
Replies
the part i'd want to pressure-test is the classifier itself, not the routing table. task classification happens before any of these branches, so whatever's deciding "is this sensitive" has to run cheap and fast at the front door, which usually means it's the least scrutinized model in the whole pipeline. a false negative there - a PII-bearing prompt gets tagged generation-heavy instead of sensitive - doesn't throw an error, it just quietly routes around your compliance boundary while GPT-4o answers it and everything downstream looks completely normal. do you validate the classifier's accuracy specifically on the sensitive bucket, separate from overall routing accuracy? that's the one class where 95% right isn't good enough, and i'd guess it's not the same number you'd quote for the other three buckets.