Your agent does something wrong. How far does it reach before anyone notices?
Two months ago I asked here how people stay aware of what their agents are doing, and @getosmo replied with something that has been quietly rearranging my roadmap since.
I had been treating "can you undo it" as the line for when an agent should stop and ask. He took that apart. Irreversibility over-fires and under-fires, he said. It over-fires on things that cannot be undone but that nobody cares about, like the agent writing a log line. It under-fires on the ones that really hurt: a mass email to 50k users, or a migration you can only roll back after downtime. "Reversible on paper, ruinous in practice."
His version is that the gate should key on reach, not on reversibility. Not "is this safe" but "how far does this reach if I'm wrong." And the part that stuck with me: the agent is the worst possible judge of its own blast radius, because it does not know there are 50k people on that list.
@tina_chhabra had pushed the same way earlier in that thread, that this gets worse the moment the agent stops writing code and starts booking meetings or sending email, where you find out after the fact. I have been sitting with both of those for two months, and they leave me with a question I still cannot answer cleanly for our own product:
If your agent gets one thing wrong today, what is the largest number of people who find out before you do?
For a coding agent that is usually one. You, reading a diff. The moment the agent sends, books, charges, replies to a customer or touches production, that number stops being one, and most of us have never actually worked out what it is. So whether you run agents or ship them to other people, which of these are you?
A. Nothing my agent does reaches past me. Genuinely low stakes, no gate needed.
B. It could reach customers, I know it, and I have not built the gate yet.
C. It stops and waits for someone to come back and look. Safe, and it quietly kills completion.
D. It emails or Slacks someone, which mostly works until the person who cares is not in that channel.
E. I built the real thing, timeouts, escalation, an audit trail, and it ate a quarter I had not planned for.
Ours, honestly: we sat on B longer than is comfortable, moved to D, and then found that delivering the question is the small part. What surrounds it is the work. Whether a timeout should mean deny or hand back to the terminal took us longer to settle than building the delivery did. So did reaching the one person who can answer
when they are not sitting in front of your product, and being able to prove afterwards who approved what.
The two I still cannot answer, and I do not think anyone has:
How do you reach the person who has to decide, when they are not looking at your product and never installed anything of yours?
And what does your system do when they simply never answer? Every answer I have heard to that one is a guess, and the guess is usually "keep waiting", which is how a job sits stalled for six hours and a customer decides your product is broken.
[Same disclosure as last time: I build Pushary, which is the ask-a-human part of this, and these days we sell it to other companies to embed as much as to people running agents in their own terminal. Obvious bias, asking anyway. The last thread taught me more about this than six months of building did.]
Which letter are you, and what did it cost you to find out? I will come back in a few weeks and post the spread.
Replies
the declared-by-the-environment fix for reach (Tyler's framing, and it's the right one) still has a gap once you look at it per-call instead of per-action. an agent that sends one email at a time in a loop looks like reach=1 to the classifier every single time, because that's genuinely what the API call is. the mass send only exists at the level of the loop, not the call, and nothing in this thread's model is watching that level. same shape as the cumulative-spend point further down for purchases, but it applies just as much to sends, writes, anything an agent can repeat. feels like reach needs a session or time-window scope in addition to a per-call one, otherwise a bad loop just walks through the gate one cleared call at a time.
@omri_ben_shoham1
The counter you would reach for first has a failure worth naming before you build it, because it does not show up until it is running. A rolling count gates call N plus one, which means the first N already went out. Against an irreversible send a threshold gate never prevents a mass send, it caps one. That is a different product, and better decided on purpose than discovered later.
Gating an aggregate before it starts needs something to declare the intended total up front, and the only thing that knows the total is the agent. Which is exactly the assertion the environment declared fix was built to remove. So the two properties do not come out of one mechanism. Environment declared reach is trustworthy and visible only per call. A declared budget is gateable in advance and only as good as the model.
You can have both if the declaration is allowed to move in one direction only. Let a stated budget raise the gate and never lower it. An agent announcing four hundred sends gets stopped on call one. An agent that says nothing falls back to the counter and gets capped rather than stopped. Worth being honest that this is a design for the buggy loop and not the adversarial one, since silence is strictly the better strategy for anything trying to get through, and the counter is what you keep for that case. Most bad loops are not trying.
On scope, the unit you count over has to be at least as trustworthy as the thing you are counting. Session ids are asserted by the client, ours included, so a session scoped budget inherits whatever the reporter is worth, and a restart resets it. Time is the one scope nothing can assert. Its weakness is pacing, and a loop slow enough to stay under a rate window is not stopped, but it now runs for hours instead of seconds. That is what a rate limit actually buys. Not prevention, a slow failure in place of a fast one, and a slow one can still be noticed while it is happening.
Where that leaves us is worse than unsolved. A policy row carries a tool pattern, a timeout, a timeout action, a delivery mode and a repo scope. Every field describes a single call, so there is no way to write your rule down at all. Meanwhile the tool actions we receive land with a session id and a timestamp, indexed on site plus session and on site plus created at. The data is already shaped for both of the scopes you named, and nothing at decision time reads it.
The part about the agent being the worst judge of its own blast radius is exactly right, and we learned that the hard way when one of our agents confidently acted on stale context — it had no idea the situation had changed because nobody told it, and why would they. Your unanswered question about what to do when nobody responds is the one the whole industry is quietly dodging — most timeout logic I've seen is just a number someone picked because they had to ship. What I haven't heard a good answer to either: how do you set that threshold differently depending on who the decision affects, not just what the decision is.
@vitalii_timonin
The part I would sit with is nobody told it, and why would they. That is not an oversight, it is the default condition for anything holding a read that nothing registered. Every other consumer of that state either polls or was designed into the change path. The agent was designed into neither, so nothing that mutated the world had any reason to know it had a subscriber.
Worth separating from confidence, because the repair looks different. It was not overconfident about the facts it had. It was correctly confident about facts that had expired. Calibration is measured against what you know, and no amount of it helps with something that stopped being true after you read it, which is why making the model hedge more does nothing here and a freshness check on the inputs is the only thing that does.
On the number, agreeing, and naming why it cannot be picked well. One value is doing two jobs that move with different variables. The duration is a claim about the approver, how long until they are plausibly reachable. The fallback is a claim about the action, what is safe with no answer at all. Bundle them and a single guess has to be right about two unrelated things, and no value is.
Which is where your question resolves for me. Who it affects should not set the number. It should select who answers, and then the number becomes a property of a person, which is measurable rather than picked. Answer latency is an observable distribution per approver. If someone's ninetieth percentile is four minutes, a sixty second timeout expires on a live and willing human every single time, and every expiry after that is a fact about your threshold rather than about them.
The other half inverts the intuition. You would think affects more people means wait longer. But waiting stops being neutral the moment the affected party is not the operator. If the decision only touches the person running the agent, the delay costs them and they chose the loop, so a long window is free. If a customer is sitting on a pending reply, the wait is being spent out of someone who never agreed to it. Stakes belong in the fallback, not the clock.
We built half of this and never connected it. Who answers is its own table, a role or a named person, re-resolved at answer time so a rule pointing at somebody who left the workspace addresses nobody instead of continuing to address them. The number lives in a different table keyed identically, on tool pattern and repo, and the routing table carries no number of its own. So we choose the person and choose the number independently and let neither inform the other. Elsewhere we compute median answer latency per person and timeout rate per person and put both on a dashboard. Every input a person derived threshold would need already exists, one join away from the thing doing the guessing.
the one-directional gate is the clever part, a declared budget can only raise the threshold, never lower it, so lying about it just gets you stopped early instead of getting away with more. the piece I don't see handled yet: what happens when two separate sessions, or two restarts of the same buggy loop, are both hitting the same real-world resource at once, say the same recipient list. each one declares 400 and gets capped at 400 individually, but if the budget lives per-session rather than per-resource, the actual total delivered is 800 and nothing looks wrong from inside either session. feels like the aggregate needs to be scoped to the thing being acted on, not the thing doing the acting, and that's a harder join given session ids are all you're asserting on the tool-action side right now
@omri_ben_shoham1
Worth separating the two failures you have stacked there, because only one of them is about trust. Session ids being client asserted is real, and it is not what breaks this. Give me a perfectly attested session id, signed, unforgeable, surviving restarts, and two honest sessions still deliver eight hundred. The scope is not weak, it is the wrong noun. Fixing the assertion buys nothing here, which is the useful part, because it means this does not get solved on the way to solving that.
The one directional trick does not survive the move either, and I would rather say so than let it stand. It works because the declarer is the thing being gated. You raise your own ceiling, so lying is self defeating. Once the number bounds a shared resource it has to be drawn down by parties who never declared it and have no reason to be bound by someone else's declaration. That is a lease, not a declaration. Advisory and self binding becomes stateful and other binding, and it is the first thing in this thread that cannot live inside a session at all.
On the join, I think you are being generous calling it hard. A join is easy once a key exists and there is no key. The resource has to be named identically by two callers who never coordinated. A recipient list is not an object with an id, it is an argument. One session built it from a query, the other from a CSV, same four hundred humans, two different strings, and nothing joins them. So a resource scoped budget needs a resource identity function per tool, and that is N functions rather than one mechanism. That is the actual reason nobody ships it, not the counting.
The version I would try is to count the leaves instead of the batch. You do not need to know that list A and list B are the same list. You need to know that one address has been sent to twice in ten minutes. The element is usually already its own canonical name, an address, an account id, a row id. The batch has no identity, the leaves do. The cost is that the tool has to enumerate its effect set before acting, which is fine for send to a list and useless anywhere the effect set is not knowable in advance.
Ours is worse than you are assuming, and then better in one place. The tool action rows are receipts. They are written from a post tool event, the target is truncated for display, nothing indexes it, and the insert sits in a try catch that logs on failure. Even with your join that reports the eight hundred, it cannot prevent it. Same bug lives in our own domain, since our targets are files and commands, and two sessions editing one file is your problem with a smaller blast radius.
Then the one place we built exactly your shape. Daily spend is a counter in Redis keyed on the site and the day, with no session anywhere in the key, so every session and every restart draws down the same number and the trip fires once under concurrency. Resource scoped, actor blind, enforced server side. It still only caps, because it counts after the spend lands. Scope and timing turn out to be separate constraints, and you have fixed the one I did not.
Thirty eight comments in and every gate described here fires on a single action. The axis nobody has put on the board yet is accumulation. Reach is being computed per action, and the thing that actually hurts is reach per window. One email to one person is low reach and passes every gate in this thread. Four hundred of them in twenty minutes is the mass email to 50k, assembled one approved step at a time, and no individual step ever looked like the thing you were guarding against. That is also why reach times signed-off-by is close but one term short. It needs a rate. A signed-off action is a licence to do that thing, not a licence to do it four hundred times, and an approval today carries no count and no clock. Two concrete moves. Gate on the running total inside a window rather than on the action. And expire the approval, so if the agent returns to the same signed-off action ten minutes later it has to ask again, because what it is doing now is a different act from the one that was approved. On your what-told-you question, the honest answer for accumulation is that nothing tells you, because every individual part looked fine on the way past. That is the whole failure mode of per-action gates.
@rabnoor_s
Accumulation is on the board, a couple of branches up, and the conclusions there are worth having rather than re-deriving. Short version of where that one landed. A running total gates call N plus one, so it caps rather than prevents, because it counts after the effect has already landed. And the hard part is not the counting, it is the naming: the running total has to be kept against a resource that two callers who never coordinated would name identically, and a recipient list is not an object with an id, it is an argument. Same four hundred humans assembled from a query and from a CSV are two different strings and nothing joins them. That is why the first of your two moves keeps getting proposed and never shipped.
Your second move is the better one and I do not think it got its due, including from you, because you listed it second. It is a different kind of fix. The running total is a control on the aggregate and it needs to name the world. An expiring approval is a control on the grant, and it needs nothing you do not already have, since the approval has an id and a timestamp for the simple reason that you issued it. You get most of the protection without ever counting anything outside your own system.
The mechanism is that re-asking is itself the throttle. Four hundred sends against a ten minute expiry is two questions, not four hundred, and you never had to know what a list is. The failure mode is precise, though, and it is worth building against from the start. It only holds if the person answering notices that this is a re-ask. If the second question looks identical to the first, the second answer is identical too, and you have converted an accumulation problem into a fatigue one. Somebody tapping yes for the fortieth time is not a control, they are a rubber stamp with latency. So the re-ask has to carry the count. Approve this again, thirty ninth time in twenty minutes, is a different question from the first one, and it is the only moment in the entire system where the aggregate can be put in front of somebody who is already looking. Which is the real answer to nothing tells you. The gate does not have to detect the accumulation. It has to display it.
Ours does the exact inverse and I would not have gone looking without this. Repeated approvals of one pattern get mined over a month, and past a small threshold with a high enough approval ratio the system offers to stop asking. Accept and it writes a rule with a zero timeout and an approve action. No count, no window, no expiry. So repetition is our evidence for trust and yours is evidence for alarm, and the only thing standing between the two readings is how wide the window is. Same statistic, opposite sign.
The inversion is cleaner than that even. Our pending question has an expiry column. The permission it produces does not. The request expires and the grant is forever, which is precisely backwards from what you are arguing.
One place we did build your shape. A scope contract ratified for the current run is the only thing in our gate allowed to override a standing auto approve, and it dies with the run. So the general form might be that a licence should be scoped to the smallest unit that has an end. A run ends. A window ends. A row does not.
Earlier this issue was causing severe token burns for the company, now over the period of time, the dev team members solved the same by making sure each project has the library ready first, UI components fixed before the AI agent starts working on things.
@bhargav5394
Yours is the only control described in this thread that actually saves tokens, and I do not think that is incidental.
Everything else here gates a call that already exists. Our hook receives the tool name and its arguments on stdin, so by the time anything has an opinion the call has been generated and paid for. A deny buys back the consequence and none of the cost, and it usually costs more than an allow, because the model then re-plans and tries again. Approval is a control on what happens. Yours is a control on what gets produced, and only the second one shows up on the bill.
The difference is where the constraint lives. A library that is already there sits in the context from the first turn, so the wrong call never forms. Our rules sit behind the wall. The agent finds out what is gated by trying it, one deny string per round trip, and hitting the first rule teaches it nothing about the second.
What makes that worse rather than merely true is that we already write a block into the agent's memory file, and the comment above that code says agents only reach for a tool when something always in their context tells them to. So we know the mechanism and we use it for the behaviour, ask before anything risky, and not for the rules, here is what is auto approved and what is not. Same file, half the payload.
Where your fix stops is the part worth marking. Pre-placing works when the right answer is knowable before the run starts. A component library qualifies. Which of four hundred sends is acceptable does not, because it depends on the state of the world at the moment of the call. Everything on that side has to be generated before it can be stopped, so the burn there is the floor rather than the failure.
coming back to this after the thread moved past my original point, and I think Dale's precision-over-presence idea actually closes it. my compare-and-increment argument was that a budget check has to be atomic or two concurrent calls both read the same stale counter and both proceed. but the real fix isn't making that check atomic, it's not handing the agent the counter at all. same move as the credential case: don't give it the value and let it decide, give it a predicate, would this exceed budget, yes or no, answered by whatever owns the counter. the race disappears because there's nothing left for the agent side to read-then-act on, same as there's no card number left to leak once you're only answering matches-on-file.
@omri_ben_shoham1
makes a lot of sense on this one, if there's something more I can help with, I'm a nudge away.
appreciate that, thanks. this thread ended up covering a lot of ground since I first jumped in, good stuff.
@omri_ben_shoham1 cheers mate
Reach is the right axis and the reason is that irreversibility over-fires and under-fires at the same time. It stops an agent writing a log line and waves through a migration you can only roll back with downtime. But there is a second thing hiding inside reach that most gates miss: it is cumulative, not per action. Every individual call can be correctly scoped and legitimate, and the damage still arrives from the four hundredth, because nothing anywhere is counting. A gate that evaluates one action at a time cannot see that by construction. The version I keep coming back to is approving a shape rather than a task: roughly this many items, roughly these targets, this blast radius. Today's run either fits the shape and proceeds, or it does not and it asks. That way the gate fires on the thing that actually changed instead of on a clock or on a category.
@rabnoor_s
right on point, we're betting on the closest interface to humans right now, lock screen.
In my case it reached exactly one document review, and I think that's the whole answer to this question: blast radius is an architecture decision, not a luck outcome.
I run a team of 25 AI marketing agents. While building an operations plan, one of them invented an employee named Teri and started assigning her support tickets. Teri does not exist. Confidence was total. Sourcing was zero.
It got caught immediately because of two boundaries I'd drawn before anything shipped. First, my agents produce drafts, not actions. Nothing publishes, sends, or spends without a human looking at it, so a wrong output is a funny story instead of an incident. Second, after Teri, I added a harder rule upstream: when an agent is missing information, it has to stop and ask instead of inventing. Slower on the rare gap, correct every time.
The uncomfortable version of this thread's question is: if your agent CAN reach production, customers, or money before a human notices, you've already decided the blast radius, you just haven't detonated yet.
@tiffany_trboyevich
agreed one hundred percent, personally believe I have a direct solution for you - https://cal.com/aadilghani/ralphnex-interviews
book a slot and happy to show you something cool
Be awesome
@aadilghani Appreciate that, Aadil! Though I should warn you the problem you’d be solving is the one I already built my answer to. That’s the product launching here on the 18th, so between now and then my calendar belongs entirely to the launch and I’m guarding it like a dragon.
Genuinely curious though, founder to founder: what’s your approach to containment? Always interested in how other people are drawing the blast radius lines. And I’m totally happy to swap notes properly after launch week. I have built some things in, just not sure it’s enough.
@tiffany_trboyevich
cheering for you from the sidelines but also see you head-on in the market soon :)
I become my own customer and check by being every type of customer which my product can have , that mostly reduces the chances and if i get some it can be fixed overnight
@dhruv_aggarwal5 valid take