Nolan Vu

Skipping the marketing version, here's how it actually works in our production deployments.

Routing logic:

- Task classification at agent input: reasoning-heavy, retrieval-heavy, generation-heavy, or sensitive

Something I keep noticing in agent-building communities: we obsess over testing the code (does the tool call work, does the API respond) but we're much looser about testing the judgment (does the agent make the right call when the input is ambiguous, adversarial, or just weird).

That gap matters more for agents than for normal software. A buggy app usually fails loudly: a crash, an error message. A poorly-designed agent fails quietly. It picks the wrong tool, hallucinates a policy, or confidently takes an action nobody wanted, and everything still looks fine in the logs.

Lately I've been trying to close that gap by using a second AI as an adversarial reviewer before anything goes near production - feeding it the agent's instructions and asking it to generate the messiest, most ambiguous, or borderline-malicious inputs it can think of, then checking how the agent would actually respond to each one.

The catch: if the AI reviewer picks up on how confident I sound about the design, it tends to go easy on me - same sycophancy problem as with product ideas, just with higher stakes here, since these agents often touch real customer data, permissions, or compliance-sensitive workflows (GDPR/HIPAA territory for a lot of us building enterprise agents).

I've noticed a pattern in almost every "how I built this" thread here lately: before touching the keyboard, people are opening a chat window with an LLM and asking it to poke holes in their idea.

Makes sense. Vibe coding has made building fast. You can go from idea to working prototype in an afternoon. But building fast doesn't mean you're building the right thing faster. If anything, it raises the cost of a wrong guess, because now you can ship the wrong product in a day instead of a month.

So I've started using AI earlier in the process to argue with me. I'll describe an idea and ask it to play the most skeptical potential customer it can, or to list every reason the idea might already be a solved problem. Sometimes that surfaces a competitor I hadn't seen. Sometimes it forces me to actually define the problem instead of jumping straight to the feature list.

Here's the part that worries me though: AI models are notoriously agreeable. If I phrase my idea with enthusiasm, I sometimes get validation back that feels a little too easy like it's mirroring my confidence rather than actually stress-testing the logic. That's dangerous, because it can create false certainty right before you commit real time to building something nobody asked for.