Your agent does something wrong. How far does it reach before anyone notices?
Two months ago I asked here how people stay aware of what their agents are doing, and @getosmo replied with something that has been quietly rearranging my roadmap since.
I had been treating "can you undo it" as the line for when an agent should stop and ask. He took that apart. Irreversibility over-fires and under-fires, he said. It over-fires on things that cannot be undone but that nobody cares about, like the agent writing a log line. It under-fires on the ones that really hurt: a mass email to 50k users, or a migration you can only roll back after downtime. "Reversible on paper, ruinous in practice."
His version is that the gate should key on reach, not on reversibility. Not "is this safe" but "how far does this reach if I'm wrong." And the part that stuck with me: the agent is the worst possible judge of its own blast radius, because it does not know there are 50k people on that list.
@tina_chhabra had pushed the same way earlier in that thread, that this gets worse the moment the agent stops writing code and starts booking meetings or sending email, where you find out after the fact. I have been sitting with both of those for two months, and they leave me with a question I still cannot answer cleanly for our own product:
If your agent gets one thing wrong today, what is the largest number of people who find out before you do?
For a coding agent that is usually one. You, reading a diff. The moment the agent sends, books, charges, replies to a customer or touches production, that number stops being one, and most of us have never actually worked out what it is. So whether you run agents or ship them to other people, which of these are you?
A. Nothing my agent does reaches past me. Genuinely low stakes, no gate needed.
B. It could reach customers, I know it, and I have not built the gate yet.
C. It stops and waits for someone to come back and look. Safe, and it quietly kills completion.
D. It emails or Slacks someone, which mostly works until the person who cares is not in that channel.
E. I built the real thing, timeouts, escalation, an audit trail, and it ate a quarter I had not planned for.
Ours, honestly: we sat on B longer than is comfortable, moved to D, and then found that delivering the question is the small part. What surrounds it is the work. Whether a timeout should mean deny or hand back to the terminal took us longer to settle than building the delivery did. So did reaching the one person who can answer
when they are not sitting in front of your product, and being able to prove afterwards who approved what.
The two I still cannot answer, and I do not think anyone has:
How do you reach the person who has to decide, when they are not looking at your product and never installed anything of yours?
And what does your system do when they simply never answer? Every answer I have heard to that one is a guess, and the guess is usually "keep waiting", which is how a job sits stalled for six hours and a customer decides your product is broken.
[Same disclosure as last time: I build Pushary, which is the ask-a-human part of this, and these days we sell it to other companies to embed as much as to people running agents in their own terminal. Obvious bias, asking anyway. The last thread taught me more about this than six months of building did.]
Which letter are you, and what did it cost you to find out? I will come back in a few weeks and post the spread.
Replies
Agent Safety & Blast Radius β FFire App Really thought-provoking breakdown. I think you've put your finger on something a lot of us are still learning the hard way β that "blast radius" is the metric that actually matters, not just technical undo-ability. To answer your question honestly: we started at B, briefly convinced ourselves we were at C, and found out the hard way we were actually at D. The gap between "it stops and asks" and "it actually reaches the right person who can answer before something happens" is much wider than I expected. Your two open questions hit home: 1. How do you reach the decision-maker when they're not in your product? 2. What happens when they just never answer? For us, the partial answer has been: don't design the system to rely on a single channel. If the agent needs approval, it should deliver that question to wherever that person actually lives β email, Slack, SMS, whatever they already use. And if they don't answer, you need an escalation path, not an infinite wait. It took us a while to build that out, and honestly we're still refining it. Interested to see the spread on your upcoming post. Curious if others have found a clean solution for the "never answer" problem or if everyone is just accepting the occasional stalled job as the cost of safety. For those of us building or running agents at any scale, we've been compiling some practical approaches to this β nothing magic, just real-world patterns for managing agent behavior at the intersection of reach and responsibility.
Pushary
@zeeshan12Β
B, then thought C, then found out D. That is the clearest description of the path anyone has given me, and I suspect it is the normal one. Nobody picks D. You arrive at it by finding out C was never really true.
On multiple channels, I agree it is necessary, and I have slowly come round to thinking it is not sufficient, for an annoying reason. Email, Slack and SMS are all places people are when they are working. The case that actually breaks things is the person who is not working. More channels raises the floor without touching the ceiling.
The other thing nobody warns you about is that going multi channel turns one question into a race. The same approval is now sitting in three places and it has to resolve exactly once, or you get two people answering differently, or someone approving a thing that already timed out.
On never answering, I do not think anyone has a clean solution, and my current theory is that it is because we keep treating it as one failure when it is three. The question never arrived. It arrived and they cannot decide. They are gone. Identical silence, and each one wants a different default. Lumping them together is why every answer to this sounds like a guess, including mine.
The one part I am fairly confident about is that the timeout has to resolve to something other than deny or wait forever. Handing the decision back to whoever started the run is usually more useful than either.
What did you land on for escalation, purely time based, or a fallback person? Happy to compare notes properly if it is useful: https://cal.com/aadilghani/coffee
C on the regulated surface, on purpose. When the wrong action is a disclosure of patient data, a stalled job is cheaper than a fast one, so we pay the completion cost knowingly. Everywhere else we are honestly D.
On your three silences, healthcare has been running an answer to that for decades and it does not require telling them apart in advance. Clinical paging separates acknowledgement from decision. Seeing the question and answering it are two distinct recorded events, so no ack means they are gone, ack with no answer means they cannot decide, and each already has a different next step without anyone guessing. An unacknowledged page walks to the next name automatically.
The part that matters for your product is who owns that ladder. You cannot solve reaching a person who never installed anything of yours, and no vendor can. The organisation has to name who is reachable at 3am and commit to it. The tool only walks the list, and the record of who acknowledged what is the thing that survives people getting comfortable, which you already said yourself.
Does Pushary let the customer define that ladder, or is the destination something you configure for them?
Pushary
@clemente_lopez1Β
The acknowledgement split is the best thing anyone has put in this thread, and it dissolves the problem rather than answering it. I was treating the three silences as something you diagnose afterwards. You are pointing out that the protocol can just make them distinguishable by construction, because seeing and deciding are separate recorded events. That is obviously right and I am not sure I would have got there on my own.
Your second point is the one I need to sit with longer. I have been describing reaching the person as our problem to solve, and you are right that no vendor can solve it. The organisation names who is reachable at 3am. The tool walks the list they gave it and keeps the record. That is a smaller claim than the one I have been making, and I think it is the true one.
To answer you straight: no, not today. Escalation for us currently means handing the decision back to whoever started the run, not walking to the next name. The destination is that person's own devices. A customer defined ladder is designed and not shipped. And acknowledgement is not a separate event from the answer, which after reading this is the piece I would build first.
The thing I would genuinely like to know from your side: what happens in clinical paging when the ladder runs out? Someone has to be last, and I have never seen a good account of what that person's silence is supposed to mean.
C, and it cost us completion exactly like you said, though not for the reason I expected.
My agents do not write code, they talk to customers. Every AI draft stops and waits for a human before it reaches anyone. What I learned is that the gate does not die from waiting. It dies the moment reading the draft stops being faster than writing the reply yourself. After that people either rubber stamp everything or route around it, and you have no gate plus extra steps.
On reach, one thing I would complicate. Our number is 1. One customer reads one wrong reply. On a reach ranking that is about the safest thing on your board, under a log line by some measures, and it is still the only action in my system I cannot take back.
So reach sorts the tail well and the middle badly. One reach-of-50k and a hundred reaches-of-one are both real and they want opposite gates. The 50k case needs one careful human once, and you can afford to make that slow. The hundred-a-day case needs a review a tired person still does properly at 5pm, which is a much less interesting problem and much harder to justify spending on.
Your near miss question, since I think you are right and I would rather confirm it than be polite. We do not find out from a system. We find out from the customer, in the next message, and they are usually annoyed. The detection channel and the damage are the same event. There was no earlier signal to catch, because nothing was broken. The reply was well written and confidently wrong about which order it was looking at.
Pushary
@jernej_jan_kocicaΒ
"No gate plus extra steps" is a failure mode I have never had a name for, and your condition for it is better than mine. I had the gate dying from waiting. You have it dying the moment reviewing costs more than doing, which is far more useful because you can actually watch that number move.
You are also right that reach sorts the middle badly, and I think the reason is that the gate is per action while your damage is cumulative. A hundred reaches of one is a reach of a hundred, but no single decision ever looks like that. Each one is individually trivial and correctly waved through. My framing misses that case completely.
Which probably means the honest answer for the hundred a day case is not a gate at all. It is sampling plus a record. Review a fraction properly instead of all of them badly, and keep enough of a trail that when a customer does complain you can find out how far the same mistake already went. Call centres have done exactly that for decades, and it is unglamorous for the same reason you gave.
Your last paragraph is the one that should worry me most, so I would rather say it than skip past it. If the detection channel and the damage are the same event, nothing I build fires. Approval gates only trigger on actions something already flagged as worth asking about, and a well written, confidently wrong reply does not trip anything at all. That is not a gap in my product, it is a limit on the whole category.
So the thing I actually want to ask: was there any signal before the customer replied? Or is the honest answer that only another person reading it would have caught it, which is the thing you just said stops being affordable?
@aadilghaniΒ There was a signal, and it is not in the text. That is what makes it worse.
The reply reads perfectly. What is wrong is which record it was grounded in, the previous order, a tracking number that was true last week. So a human reading only the draft does not catch it either. They approve it for the same reason the model wrote it, because it is a good answer to the question, and neither of them checked whether it is about the right thing.
Which means the reviewable unit was never the text. It is the text plus the source it used. Put the record next to the draft and the wrong ones become obvious in about a second, because the order number is not the one the customer is asking about. Hide the source and you have made the reviewer's job impossible while appearing to hand them control.
The cheap version of that is mechanical. Compare the record the model cited against the customer's current one, escalate only on a mismatch. That is your tripwire from the last thread, one domain over, and it does not need a model, just a comparison.
On sampling plus a record, I think you are right and the record is the bigger half. A sample finds the bad reply that happened to be in the sample. The record is what tells you the same wrong source was used forty other times, which is the actual damage and the only thing that lets you go back and fix it. It also answers what Asad said above, since the ones nobody complains about are only findable that way.
So the question back: does anything in yours carry the inputs a step used, or only what the step wants to do? That is the difference between an approval someone can actually make and a yes or no on a claim.
Pushary
@jernej_jan_kocicaΒ
You are right, and the connection is exact. That thread landed on not being able to build a tripwire out of the thing that is tripping. The reason your comparison escapes that is it is not an inference. The customer's current order sits outside the agent, in a place its blind spot cannot reach. Same principle, and yours is the cheaper instance, because it needs no judgment at all, just an equality check.
The larger thing you said I want to repeat rather than soften: the reviewable unit is the action plus what it was grounded in. Hand over the claim alone and you have moved the failure from the model to the reviewer while making it look like control. That is a fair description of most approval interfaces, mine included.
So, straight answer: only what the step wants to do. We carry the user's stated intent, the concrete action, the reason it stopped, and the diff or the full command. Every one of those describes the proposal. Nothing carries the records the step actually read to arrive at it. There is a free text context field a caller can put anything into, but no structured place for provenance and nothing that compares it against anything, which makes it exactly the yes or no on a claim you are describing.
And yes on the record being the bigger half. A sample tells you this one was wrong. The record tells you the same stale source was used forty times, which is the part you can go and fix, and the only route to the ones nobody complained about.
The design question I do not have an answer to is granularity. For your support reply the provenance is one order id. For a code change it is arguably the whole repo. Did you find it was enough to carry identifiers and let the reviewing side resolve them, or does this only work when the actual record is sitting next to the draft?
@aadilghaniΒ Identifiers were not enough, and the reason is the same economics as the gate.
If the reviewer has to click through to resolve an id, that is a fetch, and at a hundred a day nobody does it. They approve on the text instead, which is the state you were already in. A pointer does not survive contact with volume.
What works sits between an id and the whole record. Carry the resolved fields the claim depends on, inline. If the draft says your order shipped on Tuesday, show the order number, its status and that date next to it. Three fields, no navigation, and the wrong ones fail on sight because the number is not the one in the customer's message.
So for granularity I would not carry the repo. Carry the specific values or lines the change asserts something about, already resolved. And if a step cannot say which those are, that is itself worth knowing about the step.
The honest limit is that this only covers claims that are about something. A reply that is about tone, or a refactor that is about structure, has no record to sit beside it, and those stay a human judgement with no shortcut..
You are probably right that the near miss usually shows up as an annoyed human, but the version that keeps me up is the one where nobody is annoyed at all. The dangerous output is not the one that looks broken, it is the one that looks fine. Plausible and wrong generates no complaint, so the detection channel everyone here is describing, the irritated customer, only fires on the failures that happen to be visible. The quietly consistent ones sail straight through, because the absence of a complaint reads exactly like success. So the honest answer to your question is that the largest number who find out before I do is not the scary number. The scary number is the people who find out and never say anything, because nothing looked wrong enough to mention. On the letters, we are a D that keeps catching itself having been a C on paper, the same trap Zeeshan described. And Clemente's point about acknowledgement being a separate recorded event from the decision is the piece I would build first, partly because it also chips at this: if seeing the action is its own event, you at least know a human's eyes crossed it, even on the days nobody complained. Great thread.
Pushary
@asadmalik901Β
The inversion is the sharpest thing in this thread and I think it breaks my own question. I asked how many people find out before you do, which quietly assumes finding out is an event. Your version is that the number that matters is the people who found out and said nothing, and that number is not observable at all. So the metric I opened with only ever measures the failures loud enough to be measured.
The thing that ties your point to Clemente's, for me, is that we are using silence as evidence in two opposite directions at once. In the approval flow, no answer means we do not know what they thought. In the outcome, no complaint means it worked. Same absence, opposite conclusions, both guesses.
Which is why your extension of his idea is better than the original. In a system where failure is quiet, acknowledgement is the only positive signal available. Everything else is measuring an absence. Ack turns "nothing happened" into "someone looked and nothing happened", and those two states are identical today despite being nothing alike.
One caveat, since otherwise this flatters my side of it too much: a tired person will ack a plausible, confidently wrong reply and approve it. Ack proves it was looked at, not that it was seen. It moves the failure from undetected to detected and waved through. Progress, not a fix.
The only traces I can think of for the silent version are second order. The customer who does not complain but asks the same thing again a week later, or quietly stops replying. Have you found any of those that actually hold up, or is it all after the fact?
we build voice agents, so our version of this is a bit different from the send-an-email case. the gate on whether the agent CAN place a call or send an SMS is easy, that part is basically E already, permissioned, logged, rate limited. the part nobody has a clean answer for is what happens inside a call that's already live. the agent can say something that's technically within its instructions but just not what a human rep would have chosen to say in that moment, and by the time anyone reviews the transcript the customer already heard it and reacted to it. there's no pause-and-ask step mid sentence. so our real answer is closer to D, we catch it after the fact through review and adjust the prompt, and we're honest that this is weaker than we'd like for anything customer facing in real time.
Pushary
@omri_ben_shoham1Β
This is the case that breaks the assumption underneath the whole thread. Everything above quietly assumes the action has a before. A send has one. A diff has one. A live call does not, because the unit of action is the entire conversation and it is already in flight.
Which makes me think approval is the wrong concept for you rather than a missing feature. The only mid-flight intervention that exists in voice is handoff, getting a human onto the line. So the question stops being can the agent pause and ask, and becomes how fast can someone join and what makes them join. That is a latency problem, and latency problems are solvable in a way that pause-and-ask is not.
The thing you have that nobody else in this thread does: your damage signal arrives while you can still act on it. Jernej made the point further up that detection and damage are usually the same event, which is why you always find out too late. In voice they are still the same event, but it is continuous. The customer interrupting, repeating themselves, going quiet, all of that lands while the call is open. Everyone else here finds out tomorrow. You find out in four seconds.
Straight about my own side: for the mid-call part, Pushary is not the tool. It is a pre-commit thing and you have no pre-commit moment. Where it would apply is the handoff, reaching whoever can actually take the line when they are not sitting in your dashboard, which is the same reachability problem one domain over.
So, do you have anything today that can pull a human onto a live call, or is the honest answer that the call just finishes?
MonoCloud for Startups
reach is a business risk. and you're right that the agent has no idea it's about to email 50k people because nobody told it the list had 50k people on it.
the part i keep thinking about is that reach is also an identity problem. the agent didn't just act, it acted with someone's credentials, someone's access, someone's blast radius. and when it goes wrong, the audit trail usually can't tell you what the agent was actually authorised to do versus what it happened to have access to. those are different things and most systems conflate them.
honestly that's a big part of why we're building MonoCloud the way we are. scoped identity for agents so the blast radius is defined upfront, not discovered after. what do you think about this?
Pushary
@riya_pariyarΒ
The distinction between what the agent was authorised to do and what it happened to have access to is the sharpest version of this I have seen, and most audit trails genuinely cannot answer it. They record the action and the outcome and never the basis.
Scoped identity is also the mechanism for something I argued further up. The agent should not be inferring its own blast radius, because it is the worst possible judge of it. It should be a property of where it is running and what it is holding. Scoping the credential is exactly that, and it puts the boundary somewhere the agent's blind spot cannot reach.
Where I would push: scope bounds the maximum blast radius, not this one. A credential correctly scoped to send customer email still lets the agent send to fifty thousand people, because that is inside the scope. Scope answers could it ever. A gate answers should it now. Neither replaces the other, which is probably why we are both building.
The place they meet is the record. Two people in this thread arrived independently at the record being the thing that survives, one from clinical paging and one from support. Your point adds that it has to carry the basis and not just the action, and I had not been thinking about it that way.
Happy to compare notes properly if useful: https://cal.com/aadilghani/coffee
Abdullah's reply upthread is the one I'd put money on being the real gap, and it went quiet, so β same experience here, from the other side of it.
I run routines that push a few dozen outbound items a day, every one gated by me before it goes. The gate was never the weak part. The completion message was. The tool reports success because from where it sits the click landed, and what's actually on the page is truncated, or posted into the wrong composer, or went out twice. So the last step of every routine is now: reload the rendered page, find the item, confirm it exists and says what I approved, and only then write it to the log. That check has caught more real failures than the approval gate has, and it costs seconds.
Which suggests a third layer next to Tyler's two. Reach is prevention, reversibility is recovery, and neither asks whether the thing you approved became the thing that exists. An action that silently didn't happen and one that happened differently both report identically.
The other thing I'd add to your letters is rate. Every one of them describes a single action. The failure I actually design against isn't a bad item, it's a run that emits fifty individually-fine ones inside an hour β each clears a per-action gate, because each is genuinely fine, and the damage lives entirely in the count and the clock. It's also where batch approval quietly becomes the problem: approving a list of fifty is the point at which reading them stops. Reach asks how far one action travels. Nothing in the framing asks how many times.
Pushary
@narek_keshishyanΒ emdashes here are wild
@aadilghaniΒ Ha, fair. Yeah, I use Claude Code for most of what I build, and these go through it. The drafts are mine though: my thinking, my call, and nothing goes out that I haven't read line by line.
Which is on topic, I suppose. That's the gate I was describing, and you just found the tell it doesn't catch. Taking the note on the dashes.