We build agents all day, and the thing I keep noticing is that they almost never fail in the middle. They fail at the very last step, when something has to happen in the real world. Ours kept dying in the same spot: a signup form that texts a six-digit code. Everything up to that point ran fine, and then it just sat there waiting for a human to go and read a text message. I'm curious what everyone else's version of this is. Not the thing your agent does badly - the thing it does perfectly, right up until it has to stop and hand it back to you. Verification codes? A booking that only happens by phone? A supplier who answers email once a week? Something that needs an actual human voice on the other end? Mostly I want to know whether the wall sits in the same place for everyone, or whether it moves depending on what you're building.
Every multi model product ends up building a router, and the first version always optimises for cost. Cheapest model that can plausibly handle the request. It works, and then it doesn't, in a way that's hard to see. The answer comes back confident and slightly wrong, nobody complains because nothing looked broken, and they quietly stop coming back.
What I actually want from a router is consistency. Same question, same shape of answer, tomorrow as well as today. Someone who has learned what your product is bad at can work around it. Someone who gets a different quality of answer every time can't learn anything, and that's worse than being reliably mediocre.
So the rule I've landed on is that a route only changes when I can explain why to the person using it, and cost isn't a reason I can say out loud.
The open question for me is detecting the plausible but wrong case without a human reading outputs. Everything I've tried is either a second model grading the first, which shares the blind spot, or a rule that only fires on mistakes nobody was going to make. If you're routing across models and have solved this, I'd like to hear how.
Meridian is an open-source, MIT-licensed, local-first AI work journal. It runs on your device, captures work that often gets forgotten, and drafts clear worklogs and project updates for you to review and approve.
If you believe productivity tools should help people make their work visible without becoming surveillance software, please support Meridian on Product Hunt. We would also value your honest feedback on what works, what feels unclear, and what we should build next.
Dial
