We had to add a "refusal button" to our enterprise AI agent after it almost got us fined

Last quarter we deployed a customer onboarding agent for a regional bank. The demo was flawless. On day 9 in production, it nearly caused a regulatory violation.

The agent was doing exactly what we asked: helpful, fast, proactive. In a regulated environment, "helpful" apparently means filling in missing customer data with its best guess. One wrong assumption about income source and we'd have been looking at an AML flag.

That incident made us do something we originally thought was dumb. We built a very visible "I'm not sure, escalate to human" button and trained the agent to use it aggressively.

Four months later, here's what actually happened:

Autonomous resolution rate dropped from ~71% to 58%. The compliance team went from "nervous" to "this is the first AI tool we don't dread." Customer satisfaction barely moved (turns out people don't mind when an AI admits it doesn't know). And we stopped getting panicked Slack messages from risk at 11pm.

The uncomfortable truth: in BFSI and healthcare, the most dangerous thing an AI agent can be is confidently wrong.

Most vendors optimize for highest resolution rate. We've started optimizing for "lowest cost of being wrong."

Has anyone else deliberately limited their production agents? What's the hardest tradeoff you've made between performance and safety?

Feel free to share your experience below and we can discuss it further about your AI agent deployment.