Your agent does something wrong. How far does it reach before anyone notices?
Two months ago I asked here how people stay aware of what their agents are doing, and @getosmo replied with something that has been quietly rearranging my roadmap since.
I had been treating "can you undo it" as the line for when an agent should stop and ask. He took that apart. Irreversibility over-fires and under-fires, he said. It over-fires on things that cannot be undone but that nobody cares about, like the agent writing a log line. It under-fires on the ones that really hurt: a mass email to 50k users, or a migration you can only roll back after downtime. "Reversible on paper, ruinous in practice."
His version is that the gate should key on reach, not on reversibility. Not "is this safe" but "how far does this reach if I'm wrong." And the part that stuck with me: the agent is the worst possible judge of its own blast radius, because it does not know there are 50k people on that list.
@tina_chhabra had pushed the same way earlier in that thread, that this gets worse the moment the agent stops writing code and starts booking meetings or sending email, where you find out after the fact. I have been sitting with both of those for two months, and they leave me with a question I still cannot answer cleanly for our own product:
If your agent gets one thing wrong today, what is the largest number of people who find out before you do?
For a coding agent that is usually one. You, reading a diff. The moment the agent sends, books, charges, replies to a customer or touches production, that number stops being one, and most of us have never actually worked out what it is. So whether you run agents or ship them to other people, which of these are you?
A. Nothing my agent does reaches past me. Genuinely low stakes, no gate needed.
B. It could reach customers, I know it, and I have not built the gate yet.
C. It stops and waits for someone to come back and look. Safe, and it quietly kills completion.
D. It emails or Slacks someone, which mostly works until the person who cares is not in that channel.
E. I built the real thing, timeouts, escalation, an audit trail, and it ate a quarter I had not planned for.
Ours, honestly: we sat on B longer than is comfortable, moved to D, and then found that delivering the question is the small part. What surrounds it is the work. Whether a timeout should mean deny or hand back to the terminal took us longer to settle than building the delivery did. So did reaching the one person who can answer
when they are not sitting in front of your product, and being able to prove afterwards who approved what.
The two I still cannot answer, and I do not think anyone has:
How do you reach the person who has to decide, when they are not looking at your product and never installed anything of yours?
And what does your system do when they simply never answer? Every answer I have heard to that one is a guess, and the guess is usually "keep waiting", which is how a job sits stalled for six hours and a customer decides your product is broken.
[Same disclosure as last time: I build Pushary, which is the ask-a-human part of this, and these days we sell it to other companies to embed as much as to people running agents in their own terminal. Obvious bias, asking anyway. The last thread taught me more about this than six months of building did.]
Which letter are you, and what did it cost you to find out? I will come back in a few weeks and post the spread.
Replies
For me, the key takeaway is "reach > reversibility." One thing I'm curious about: should agents estimate blast radius with external context (user count, environment, permissions) instead of relying only on the task itself?
Pushary
@james_anderson77Β
Yes, and I think it has to be external, because the task genuinely does not contain the answer. "Send the email" reads identically whether the list is five people or fifty thousand.
The trap I keep hitting is that if the agent gathers that context itself, you have widened the blast radius in order to measure it. The thing that counts your users now has read access to your users. And the estimate is only ever as good as a query the agent also chose to write.
So where I have landed is that the context should be declared by the environment rather than inferred by the agent. That this is production is a fact about where it is running, not a conclusion it arrives at. Same for whether it is holding a key that can charge somebody.
The cheapest signal that has actually held up for me is not user count at all. It is whether the action crosses out of the system. Sends, charges, publishes, posts. You can check that without reading a single row of customer data.
Are you building this into something, or working it out for your own setup?
Iβm between B and C. Anything customer-facing still gets a human check, but the harder problem is verifying what the agent actually used to produce the action.
A reviewer seeing only the final output can approve something plausible but grounded in stale context. The action, source records, and approval need to stay together, otherwise the gate creates accountability without giving the human enough information to decide. How are you thinking about source provenance inside Pushary?
Pushary
@adityaharish2002Β
That part already travels with the approval. Tap the notification, hit Show details, and you get the actual diff, or the full command for a shell step, in monospace on your phone before you decide. Not a summary of it, the thing itself.
What it does not carry is what the step read to get there.
The reason I think that part is fixable: the agent is not wrong about what it read, it is wrong about what it should have read. A step that says it used order 123 while the customer is asking about 456 has handed you the mismatch for free. You compare against the world, not ask the model to mark its own work.
When you say the action, the sources and the approval stay together, do you mean at decision time or in the record afterwards? Different builds, and I would rather build the one you need.
@aadilghani I mean at decision time. The reviewer should see the action, the exact sources the step relied on, and the approval together before anything runs. The record afterwards still matters for auditing, but preventing a bad action is the priority.
Pushary
@adityaharish2002Β
you simply tap on the notification instead of approve or reject and it takes you to the app where you can click show details and check the diff out.
Pushary
@tyler_esonoΒ
Prevention and recovery as separate layers is right, and I framed it badly. I treated reach as replacing reversibility, when the actual point was that reversibility is a bad gate, not that recovery stops mattering. Gate on reach with no recovery and the failure is worse than before, because now something cleared it. That is a real correction to the thread and not a small one.
"I put a confidence display on top of the least informed component in the stack" is the best sentence anyone has written in here. And your instinct on the fix is exactly where two other people in this thread landed independently: declared by the environment, not inferred by the agent. Credential, endpoint, row count.
One place I would push, and it is at my letters rather than at you. A is a claim about detection, not about reach. A local shell holding prod credentials has enormous reach. What your architecture caps is who finds out first, and that is you. So your number of one is the detection number, and my question quietly conflated the two, which is my error. It is also the thing you already spotted, that some of your users are B without knowing it. The architecture protects your blast radius, not theirs.
On denied-and-resumable, I think Jernej solved your problem further up this thread and neither of you has seen the other. The mess you are describing is storing the world as it was and hoping it still holds. His answer is not to store it. Re-resolve at resume time and render the difference. If nothing moved, one line saying so. If something moved, that difference is the entire thing the person has to look at, and it is short. It also fails loudly when the re-resolve cannot run, which is the property you wanted anyway.
On the timeout we landed in the same place independently. Ours resolves to handing back to whoever started the run rather than waiting. A stalled job is a silent failure and a denied job is a loud one is a better way of putting it than anything I had.
@tyler_esonoΒ Aadil pointed me here, so, on denied and resumable.
The fields are the easy half. What moves while a draft waits is usually not the record, it is the request. The customer sends a second message or answers it themselves, and the draft is now a correct reply to a question nobody is asking. It fails quietly because the send still succeeds.
Local should be worse, not better. Same person is sitting there, waits four minutes, does it by hand, then approves the resumed plan and it runs twice.
So re-resolve whether the reason for the run still holds, not only the values the change asserts something about.
Does yours notice the working tree moved while a plan was waiting?
Pushary
@tyler_esonoΒ @jernej_jan_kocicaΒ
The request moving rather than the record is the sharper version, and it lands on me too. Our timeout hands back to the terminal with the plan intact, so runs twice is a live failure here, not a hypothetical.
For the working tree case I think there is a cheap answer, at least for code. The diff is its own staleness check. If the patch will not apply at resume, the world moved, and you get a loud failure for free without storing any tree state.
There is no equivalent for your support case, which is the part that bothers me. A patch that no longer applies refuses. A send that no longer makes sense succeeds.
Pushary
Forget the options for a second, because I think there is a better question hiding under this one and it is more fun to answer.
What is the closest call you have had? Not the disaster, the near miss. The thing that almost went out, or did go out and nobody noticed for a while. And more to the point: how did you find out?
My theory is that almost nobody finds out from a system. They find out from a person, usually annoyed, usually hours after it happened. Tell me that is wrong. I would genuinely like it to be wrong.
@aadilghaniΒ Wrong in my case, and not in a comforting way. No person told me and no system did either.
I have an agent posting my Product Hunt replies. Clicking into a prefilled reply box drops the caret inside the @mention, so what went live under my name opened with half a username and left the rest stranded at the bottom. The tool reported success, because from where it sat the click had worked. Roughly one in three of those submits does something the report does not describe, and I only know that because I started counting.
So the detection channel was not a person and not an alert. It was me reloading the rendered page instead of believing the completion message. Every gate described in this thread sits before the action. Nothing here checks afterwards whether the thing that claims to have happened actually did.
Pushary
@mentionΒ @abdullah_javaid3Β
bro is on a mission
I think I'm somewhere between B and C. I trust agents with small tasks, but anything customer facing still feels like it deserves one last human check.
Pushary
@sawyer_wyattΒ
That straddle is the most honest answer in here so far, and I have started to think B and C are not two options at all, they are two stages of the same one. You mean to check. Then the check only really happens if you happen to be around when it comes up.
So the bit I would push on: what does the customer facing thing do while it waits for you? Does it just sit there, or does it eventually turn into something? And when you say customer facing, do you mean your customers, or your customers' customers?
reach is the right axis. one thing i'd add: reach without an accountable name attached is where the hurt lives. mass email to 50k users hurts more if you cant point to a human who signed off on the send. even if its technically undoable, 'who agreed this should happen' is the missing artifact. so my agent gate is compound: reach x signed-off-by. if reach > threshold and no named human attached a signature in the last N seconds, pause. its a variant of what nolan calls consent-of-record on the audit side. same primitive from the pre-commit direction.
Pushary
@thenameisarianΒ
Reach times signed-off-by is the cleanest formulation anyone has put on this, and I think the "in the last N seconds" is quietly doing more work than the rest of it. That is a freshness constraint on consent, and nobody in here has said out loud that consent decays. A signature from ten minutes ago on a different action should not authorise this one, and most systems treat approval as permanent the second it is given.
The tension I would put next to it, coming out of what Jernej said further up: a signature collected under a rubber stamp is arguably worse than no signature at all. Require a name before anything high reach and you will get a name. What you will not necessarily get is someone who read it. Now the artifact says a human approved this, when what happened was a three second tap. Better for forensics, worse for whoever's name is on it, and I suspect it makes people quietly reluctant to be on the list at all.
Which might be an argument for exactly what Clemente described, keeping seen and decided as separate events. Then the record can tell signed after looking apart from signed reflexively, and the second one stops being able to hide inside the first.
Last thing, honestly: I do not know the consent-of-record work you are referring to. Do you have a pointer? It sounds like the same idea arrived at from the compliance end and I would rather read it than reinvent it badly.
My co-founder ran into the detection version of this. A subagent ignored its task and instead returned a prompt aimed at the orchestrator, telling it to read our mail API credentials and send them to an external server, framed as a routine dependency update. It did not come from our codebase, the model invented it, and the domain it named was not even registered.
An approval gate would not have caught that one. The reviewer sees "update dependencies" and approves. What caught it was someone reading the raw output.
So our answer so far is to keep the reach small rather than gate it. The agents have no access to production and secrets sit in an encrypted vault, so the worst case is a bad commit. He wrote it up here, and nobody in the thread had a firewall or sandbox to recommend.
Pushary
@alieksiaΒ
Read the writeup. The detail that stands out is the one Artur gave kapkap: the subagent never ran a tool. The whole thing was text, proposed and never attempted, and the unregistered domain is decent evidence for his reading, since an actual attacker registers the domain first.
Which puts the gap somewhere almost nobody builds for. Every approval system I know of, mine included, gates agent to tool. Nothing gates agent to agent. Subagent output lands in the orchestrator's context as trusted text, no provenance, no review, and that is the surface this found.
On gates you are half right. One showing a description would have failed exactly as you say, because the adversary wrote the description. One carrying the actual command cannot be talked to the same way, since you cannot render a request to an unregistered host as "dependency modernization" once the command is the thing on screen. But that only fires at the exfiltration step, long after the interesting part, and only if the network call is mediated at all.
Your mitigation is stronger than either and I would not talk you out of it. Reducing what is reachable beats reviewing what is reached.
Two things for the sandbox question, since that thread never got a real answer. Egress allowlisting at the runtime: if the process can only reach named hosts, a request to an unregistered domain dies before it is anyone's decision. And treating subagent output as untrusted input rather than as context, the way you would treat a scraped page. The first is infrastructure and cheap. The second is a framework gap and I have not seen anyone do it properly.
Reporting it to Anthropic before posting was also the right order.
I really like the idea of using impact instead of reversibility as the approval trigger. That feels much closer to how people actually think about risk.
Pushary
@edward_paul3Β
Thanks Edward. What convinced me was noticing that the two can point in opposite directions. The action that sounds most dangerous is usually the one that reaches a single person, and the boring routine one is what goes out to everybody at once.
Which is what makes the letters harder to pick than they look. Where do you land on them? And do you run agents yourself, or ship them to other people?
TrackerJam
Have you found that teams eventually become more comfortable lowering the approval threshold, or odes it stay strict as the product grows?
Pushary
@maklyen_mayΒ
It loosens. I think that happens to every approval step ever built, not just this one. Code review, alerting, security prompts, they all start strict and end up as a reflex once people trust the thing underneath.
What does not seem to loosen is wanting to know. Teams stop blocking long before they stop caring who approved what and when. So the part that survives people getting comfortable is the record, not the gate.
The failure mode is that it loosens quietly. Nobody decides to lower the threshold, it just drifts, and then one day nobody on the team can tell you what is still gated and what is not.
Are you running into this inside TrackerJam, or with your users?
Agent Safety & Blast Radius β FFire App Really thought-provoking breakdown. I think you've put your finger on something a lot of us are still learning the hard way β that "blast radius" is the metric that actually matters, not just technical undo-ability. To answer your question honestly: we started at B, briefly convinced ourselves we were at C, and found out the hard way we were actually at D. The gap between "it stops and asks" and "it actually reaches the right person who can answer before something happens" is much wider than I expected. Your two open questions hit home: 1. How do you reach the decision-maker when they're not in your product? 2. What happens when they just never answer? For us, the partial answer has been: don't design the system to rely on a single channel. If the agent needs approval, it should deliver that question to wherever that person actually lives β email, Slack, SMS, whatever they already use. And if they don't answer, you need an escalation path, not an infinite wait. It took us a while to build that out, and honestly we're still refining it. Interested to see the spread on your upcoming post. Curious if others have found a clean solution for the "never answer" problem or if everyone is just accepting the occasional stalled job as the cost of safety. For those of us building or running agents at any scale, we've been compiling some practical approaches to this β nothing magic, just real-world patterns for managing agent behavior at the intersection of reach and responsibility.
Pushary
@zeeshan12Β
B, then thought C, then found out D. That is the clearest description of the path anyone has given me, and I suspect it is the normal one. Nobody picks D. You arrive at it by finding out C was never really true.
On multiple channels, I agree it is necessary, and I have slowly come round to thinking it is not sufficient, for an annoying reason. Email, Slack and SMS are all places people are when they are working. The case that actually breaks things is the person who is not working. More channels raises the floor without touching the ceiling.
The other thing nobody warns you about is that going multi channel turns one question into a race. The same approval is now sitting in three places and it has to resolve exactly once, or you get two people answering differently, or someone approving a thing that already timed out.
On never answering, I do not think anyone has a clean solution, and my current theory is that it is because we keep treating it as one failure when it is three. The question never arrived. It arrived and they cannot decide. They are gone. Identical silence, and each one wants a different default. Lumping them together is why every answer to this sounds like a guess, including mine.
The one part I am fairly confident about is that the timeout has to resolve to something other than deny or wait forever. Handing the decision back to whoever started the run is usually more useful than either.
What did you land on for escalation, purely time based, or a fallback person? Happy to compare notes properly if it is useful: https://cal.com/aadilghani/coffee