What should your AI agent never be allowed to try on its own?
by•
been leaving comments on a bunch of the AI agent launches this week and noticed the same question keeps coming up in different forms - not "is the agent good enough" but "what's the one thing it's flatly not allowed to decide for itself, no matter how confident it sounds." voice agents that hand off emergencies to a human, coding agents that won't push to main, ones that won't touch payment or lock systems without a human sign-off.
Curious what your actual hard line is, the one thing you'd never let an agent attempt unsupervised even if it swears it knows what it's doing. and just as interesting - has an agent ever actually tried to cross that line on its own, or is it still theoretical for most of you?
13 views
Replies
for us (voice AI in hospitality/services) the hard line is anything that becomes irreversible in the same breath it's said - booking, printing, sending, charging. there's a very concrete example of this exact failure a few threads back: an assistant hedged when asked if they print diplomas, got asked again more directly, and the second time said yes - file went to print before anyone caught it. no scope decision was made, no confirmation step, just a hedge turning into a confident yes under a bit of pressure. that's the thing I'd never let an agent do unsupervised - let a hedge become a commitment just because someone asked twice. and to your second question, yes it's not theoretical, we've seen agents affirm something on a call that then gets acted on in the same utterance, so there's no turn boundary to catch it after the fact like you'd get in chat
Hard line for us: an agent should not turn an uncertain customer signal into a committed product promise or shipped change without a human making the call. The scary failure isn’t only booking or charging; it’s when a plausible summary quietly becomes a backlog item, then an agent prompt, and the original customer reason is gone. We want the human to see the source, intended outcome, and evidence that would change the decision before anything crosses that line. Confidence is not approval; a confident hedge is still a hedge. Curious how others make that context visible at the moment of approval.
>=PlayingFild
Never let it destroy something the user can't get back. And assume the human is always right, even when the model disagrees. That disagreement is a good data point, not an error.
Mine did cross it this week. One code path was exempt from a guard every other path respected, so a user action closed a tab that should have been protected. The failure was because of an exemption granted for a good reason in one place became the issue for another.