Your agent does something wrong. How far does it reach before anyone notices?
Two months ago I asked here how people stay aware of what their agents are doing, and @getosmo replied with something that has been quietly rearranging my roadmap since.
I had been treating "can you undo it" as the line for when an agent should stop and ask. He took that apart. Irreversibility over-fires and under-fires, he said. It over-fires on things that cannot be undone but that nobody cares about, like the agent writing a log line. It under-fires on the ones that really hurt: a mass email to 50k users, or a migration you can only roll back after downtime. "Reversible on paper, ruinous in practice."
His version is that the gate should key on reach, not on reversibility. Not "is this safe" but "how far does this reach if I'm wrong." And the part that stuck with me: the agent is the worst possible judge of its own blast radius, because it does not know there are 50k people on that list.
@tina_chhabra had pushed the same way earlier in that thread, that this gets worse the moment the agent stops writing code and starts booking meetings or sending email, where you find out after the fact. I have been sitting with both of those for two months, and they leave me with a question I still cannot answer cleanly for our own product:
If your agent gets one thing wrong today, what is the largest number of people who find out before you do?
For a coding agent that is usually one. You, reading a diff. The moment the agent sends, books, charges, replies to a customer or touches production, that number stops being one, and most of us have never actually worked out what it is. So whether you run agents or ship them to other people, which of these are you?
A. Nothing my agent does reaches past me. Genuinely low stakes, no gate needed.
B. It could reach customers, I know it, and I have not built the gate yet.
C. It stops and waits for someone to come back and look. Safe, and it quietly kills completion.
D. It emails or Slacks someone, which mostly works until the person who cares is not in that channel.
E. I built the real thing, timeouts, escalation, an audit trail, and it ate a quarter I had not planned for.
Ours, honestly: we sat on B longer than is comfortable, moved to D, and then found that delivering the question is the small part. What surrounds it is the work. Whether a timeout should mean deny or hand back to the terminal took us longer to settle than building the delivery did. So did reaching the one person who can answer
when they are not sitting in front of your product, and being able to prove afterwards who approved what.
The two I still cannot answer, and I do not think anyone has:
How do you reach the person who has to decide, when they are not looking at your product and never installed anything of yours?
And what does your system do when they simply never answer? Every answer I have heard to that one is a guess, and the guess is usually "keep waiting", which is how a job sits stalled for six hours and a customer decides your product is broken.
[Same disclosure as last time: I build Pushary, which is the ask-a-human part of this, and these days we sell it to other companies to embed as much as to people running agents in their own terminal. Obvious bias, asking anyway. The last thread taught me more about this than six months of building did.]
Which letter are you, and what did it cost you to find out? I will come back in a few weeks and post the spread.
Replies
This is such a good way to frame agent safety how far does it reach if I’m wrong? feels much more practical than simply asking whether an action is reversible.
I am especially curious about the timeout problem: if the right person doesn’t respond, how do you balance preventing a bad action with keeping the agent from becoming a glorified waiting room?
Pushary
@ethan_walker9
The timeout problem is the one I'm least done thinking about, so I'll give you the honest state of it rather than a clean answer, because I don't think anyone has the clean answer yet.
The trap is that both defaults are wrong. Fail-closed (deny on no-answer) is safe but turns the agent into the waiting room you're describing, work stalls, and a customer decides your product is broken while a job sits blocked for six hours. Fail-open (proceed on no-answer) keeps things moving and quietly defeats the entire point of the gate, because the actions most likely to time out are the ambiguous, high-stakes ones nobody wanted to rubber-stamp, and those are exactly the ones you'd be auto-approving. So "keep waiting" and "just proceed" are both failure modes, just failures pointed in opposite directions.
The reframe that's helped me most: the timeout default shouldn't be global, it should key on the same reach question that decides whether to gate at all. If an action reaches one person and is cheap to undo, timing out to proceed is fine, the cost of being wrong is low, so don't hold up the work. If it reaches 50k people or touches production, timing out has to fail closed, because the whole reason it was gated is that being wrong is expensive, and a missing human doesn't make it cheaper. So the same reach × authority calculation that decides whether to ask also decides what silence means. High reach, silence means stop. Low reach, silence means go. One principle, both decisions.
But the sharper move, which came out of another thread here, is that fail-closed shouldn't mean "freeze forever," it should mean "expire the premise and re-derive from current state." The reason a job sits stalled for six hours isn't the timeout, it's that the agent is holding a stale plan waiting to resume it. If the timeout instead invalidates the decision and forces the agent to recompute against the world as it is now, you avoid both the glorified-waiting-room and the resume-on-a-stale-premise failure. The agent doesn't wait indefinitely, it abandons the specific blocked action and re-derives, which might mean the action is no longer needed, or needs a different human, or the world moved and it's moot.
And the part people skip: the timeout is also a routing signal, not just a decision. If the right person didn't answer, the honest response often isn't "decide without them," it's "escalate to someone who can" or "the person who cares wasn't in the channel I pinged." A no-answer frequently means you asked the wrong person or the right person somewhere they weren't looking, and treating it purely as a proceed/deny decision misses that the real fix is reaching a different human before you fall back to any default at all.
So my current honest position: timeout default scales with reach, fail-closed means re-derive-not-freeze, and a no-answer should try re-routing before it resolves to any automatic decision. What I don't have clean is how long to wait before each of those kicks in, because too short floods people with escalations and too long is the six-hour stall, and I suspect that number can't be set globally either, it's per-action-reach again.
Genuine question back, since you're clearly thinking about it: when you imagine the timeout firing, is your worry more the false-proceed (it did something bad because nobody answered) or the false-freeze (it stalled something harmless because it was being overly cautious)? Because which one you're more afraid of basically sets your default, and I've found people split hard on it depending on whether they've been burned by a runaway action or by a dead product, and that split might be the real answer, that the default is a function of what you're most scared of, not a universal.
D, and the number came out bigger than i had it modelled. ours reaches store owners by email, so the blast radius is a list of real businesses forming a first impression of the company, not something that stays with me. what changed my gate was realising the agent cannot tell a draft from a send. it has no way of knowing that one of those is recoverable in ten seconds and the other one spends a first impression permanently, on someone we dont get to re-approach. so we gate on permanence now rather than reach. anything reversible inside a minute runs unattended. anything that spends a first impression waits for a person.
The dimension that mattered for us was not reversibility or blast radius, it was whether the agent is sure.
I run price and stock checks across a product catalogue. A check that comes back empty because the page was blocked looks exactly like a real change, so the dangerous write is not the destructive one, it is the confident one made on bad input. Reversible on paper, and nobody notices for weeks because nothing errored.
So the gate we ended up with is: a failed read is not allowed to become data. It writes nothing, only the last-checked timestamp moves, which keeps attempted and learned as two separate fields. Availability is derived from positive evidence only, so a page that does not say in stock counts as unread rather than sold out.
That covers a whole class of quiet damage without asking a human anything, which is the part I like. The agent does not need permission to not write.
Disclosure, same as everyone else here: I build and run affiliate storefronts for people, so this failure mode is my day job rather than a thought experiment.