Would you trust an AI agent that gets 98% of decisions right?
I have been thinking about how we decide whether an AI agent is actually ready for production.
Imagine you build an agent that successfully handles 98 out of 100 customer requests. On paper, that sounds impressive. Most teams would probably consider it a success.
But what happened in the other two cases?
There is a big difference between an AI giving an incorrect answer and an AI accidentally issuing a refund, cancelling a subscription, or changing someone's account details.
Both count as failures, but their impact is completely different.
This is where I think traditional accuracy metrics start becoming less useful. We need to measure not just whether an agent completes a task, but whether it knows when to stop, verify information, or involve a human.
I am starting to think that an agent with 90% task completion and reliable safeguards might be more valuable than one with 98% completion but unpredictable failures.
For founders and builders deploying AI agents, how are you deciding when an agent is safe enough for production?
Are you using specific benchmarks, human approvals, limited permissions, or something else?
Replies
I don't decide how accurate the agent has to be. I decide where its 2% is allowed to land, and I design the product so that it lands somewhere cheap. In my app the model's job is to write and sing a song. When it gets one wrong, the user hears a bad song and generates another, which is annoying and fully reversible. The model has no path to anything that isn't: the credits are written only by the backend after a payment event, the security rules forbid the client and the model from touching them, and a support reply the model drafts sits in a queue until I've read and edited it. So the agent could be 80% right and still be safe to ship, because none of the failures can issue a refund or cancel anything. The same thinking applies to the coding agent that builds the app. It may deploy the backend and the admin panel on its own, those are routine and reversible. It may not publish a store build or deploy during an incident without a word from me, and the rules say so in writing. So to your question: not a benchmark, and not approvals on everything either. A short list of actions that are irreversible, a hard wall around them, and freedom everywhere else. An agent that knows when to stop is nice. An agent that can't reach the dangerous button is better, because it doesn't need to know.