Every multi model product ends up building a router, and the first version always optimises for cost. Cheapest model that can plausibly handle the request. It works, and then it doesn't, in a way that's hard to see. The answer comes back confident and slightly wrong, nobody complains because nothing looked broken, and they quietly stop coming back.
What I actually want from a router is consistency. Same question, same shape of answer, tomorrow as well as today. Someone who has learned what your product is bad at can work around it. Someone who gets a different quality of answer every time can't learn anything, and that's worse than being reliably mediocre.
So the rule I've landed on is that a route only changes when I can explain why to the person using it, and cost isn't a reason I can say out loud.
The open question for me is detecting the plausible but wrong case without a human reading outputs. Everything I've tried is either a second model grading the first, which shares the blind spot, or a rule that only fires on mistakes nobody was going to make. If you're routing across models and have solved this, I'd like to hear how.
We're running a one-day AI hackathon in Madrid on Saturday Sept 12. Deliberately small: 20 curated spots, engineers and founders, individual format, something working by the end of the day. Then food and beers while people demo.
The plan is to repeat it every month or so, so I care more about getting the format right than about this one edition.
Two things I keep going back and forth on:
Individual vs teams. Individual means everyone ships something and nobody hides. Teams mean more ambitious projects but a worse experience for whoever gets the bad group. I've landed on individual for a one-day format, but I'm not sure.
