Pushary
Control panel for AI agents on your lock screen
Start new thread

Your agent does something wrong. How far does it reach before anyone notices?

by
•

Two months ago I asked here how people stay aware of what their agents are doing, and replied with something that has been quietly rearranging my roadmap since.

I had been treating "can you undo it" as the line for when an agent should stop and ask. He took that apart. Irreversibility over-fires and under-fires, he said. It over-fires on things that cannot be undone but that nobody cares about, like the agent writing a log line. It under-fires on the ones that really hurt: a mass email to 50k users, or a migration you can only roll back after downtime. "Reversible on paper, ruinous in practice."

His version is that the gate should key on reach, not on reversibility. Not "is this safe" but "how far does this reach if I'm wrong." And the part that stuck with me: the agent is the worst possible judge of its own blast radius, because it does not know there are 50k people on that list.

had pushed the same way earlier in that thread, that this gets worse the moment the agent stops writing code and starts booking meetings or sending email, where you find out after the fact. I have been sitting with both of those for two months, and they leave me with a question I still cannot answer cleanly for our own product:

If your agent gets one thing wrong today, what is the largest number of people who find out before you do?

For a coding agent that is usually one. You, reading a diff. The moment the agent sends, books, charges, replies to a customer or touches production, that number stops being one, and most of us have never actually worked out what it is. So whether you run agents or ship them to other people, which of these are you?

A. Nothing my agent does reaches past me. Genuinely low stakes, no gate needed.

B. It could reach customers, I know it, and I have not built the gate yet.

C. It stops and waits for someone to come back and look. Safe, and it quietly kills completion.

D. It emails or Slacks someone, which mostly works until the person who cares is not in that channel.

E. I built the real thing, timeouts, escalation, an audit trail, and it ate a quarter I had not planned for.

Ours, honestly: we sat on B longer than is comfortable, moved to D, and then found that delivering the question is the small part. What surrounds it is the work. Whether a timeout should mean deny or hand back to the terminal took us longer to settle than building the delivery did. So did reaching the one person who can answer

when they are not sitting in front of your product, and being able to prove afterwards who approved what.

The two I still cannot answer, and I do not think anyone has:

How do you reach the person who has to decide, when they are not looking at your product and never installed anything of yours?

And what does your system do when they simply never answer? Every answer I have heard to that one is a guess, and the guess is usually "keep waiting", which is how a job sits stalled for six hours and a customer decides your product is broken.

[Same disclosure as last time: I build Pushary, which is the ask-a-human part of this, and these days we sell it to other companies to embed as much as to people running agents in their own terminal. Obvious bias, asking anyway. The last thread taught me more about this than six months of building did.]

Which letter are you, and what did it cost you to find out? I will come back in a few weeks and post the spread.

1.4K views

Add a comment

Replies

Best

Reach is the right axis and the reason is that irreversibility over-fires and under-fires at the same time. It stops an agent writing a log line and waves through a migration you can only roll back with downtime. But there is a second thing hiding inside reach that most gates miss: it is cumulative, not per action. Every individual call can be correctly scoped and legitimate, and the damage still arrives from the four hundredth, because nothing anywhere is counting. A gate that evaluates one action at a time cannot see that by construction. The version I keep coming back to is approving a shape rather than a task: roughly this many items, roughly these targets, this blast radius. Today's run either fits the shape and proceeds, or it does not and it asks. That way the gate fires on the thing that actually changed instead of on a clock or on a category.

 

right on point, we're betting on the closest interface to humans right now, lock screen.

In my case it reached exactly one document review, and I think that's the whole answer to this question: blast radius is an architecture decision, not a luck outcome.

I run a team of 25 AI marketing agents. While building an operations plan, one of them invented an employee named Teri and started assigning her support tickets. Teri does not exist. Confidence was total. Sourcing was zero.

It got caught immediately because of two boundaries I'd drawn before anything shipped. First, my agents produce drafts, not actions. Nothing publishes, sends, or spends without a human looking at it, so a wrong output is a funny story instead of an incident. Second, after Teri, I added a harder rule upstream: when an agent is missing information, it has to stop and ask instead of inventing. Slower on the rare gap, correct every time.

The uncomfortable version of this thread's question is: if your agent CAN reach production, customers, or money before a human notices, you've already decided the blast radius, you just haven't detonated yet.

 

agreed one hundred percent, personally believe I have a direct solution for you -

book a slot and happy to show you something cool

Be awesome

 Appreciate that, Aadil! Though I should warn you the problem you’d be solving is the one I already built my answer to. That’s the product launching here on the 18th, so between now and then my calendar belongs entirely to the launch and I’m guarding it like a dragon.

Genuinely curious though, founder to founder: what’s your approach to containment? Always interested in how other people are drawing the blast radius lines. And I’m totally happy to swap notes properly after launch week. I have built some things in, just not sure it’s enough.

 

cheering for you from the sidelines but also see you head-on in the market soon :)

I become my own customer and check by being every type of customer which my product can have , that mostly reduces the chances and if i get some it can be fixed overnight

 valid take

Irreversibility is the real line, not undo-ability.

 quite a statement tbh

adding a dimension to reach that doesn't route through people at all. once an agent's output is another agent's input, not a human reading it, an MCP-style tool call feeding straight into the next hop, reach stops being about how many people find out and becomes about how many automated consumers act on it before any human does. a wrong number my agent produces, read by a person, might get caught on sight. the same wrong number consumed by another company's agent as ground truth gets acted on, and possibly re-published as that agent's own output, before anyone with eyes on it exists at all. the blast radius compounds per hop, and every hop after the first has zero chance of catching the original error, because none of them were there when it happened, they just inherited a "trusted" input. curious if anyone building for the tool-calling case has a version of the destination check for a consumer they don't operate and can't instrument.

 this could be useful for low blast radius tasks and non mission-critical products ideally

 fair, and that scoping is probably the honest answer for where a destination check works today. what I keep getting stuck on is the other half: for the high blast radius, mission critical case, the one where checking actually matters most, what are teams actually doing right now, if anything? my guess is mostly nothing, just trust boundaries drawn at the org/API level and hoping the upstream agent behaved, which is a pretty thin guarantee once money or production state is on the other end.

Mine, and it agrees with Abdullah: no person and no system.

My browser agent decided whether an action worked by folding the returned string and testing it for phrases like "could not" and "stopped because". The password refusal reads "Candoa does not enter credentials." That matches none of them, so a refusal was reported upstream as executed and the model kept going, believing the field was filled. The block held. The accounting of the block did not.

I found it by driving it myself. Nothing in the path could have told me, because the layer that would report the failure was the layer producing the wrong answer.

   fair, what would you have preferred in an ideal world?

@aadilghani I don't think TTL fixes it, and here's why from the actual case. The stale premise wasn't a fact that decayed over time and needed a refresh timer - it was a wrong value that got written down once (a profile slug) and then every later check just read the stored copy instead of the live source. A TTL on that would still be re-reading the same wrong copy, just less often, until the timer happened to line up with someone finally going upstream.

The fix that actually worked wasn't "recheck more often," it was "stop storing the fact at all, derive it fresh from something that can't go stale" - in my case, following the account link from the live logged-in session instead of trusting a saved slug. That's not expiry, it's removing the cache entirely for anything that's cheap enough to re-derive on every check. TTL is the right tool when re-deriving is expensive and you're accepting some staleness on purpose. It's the wrong tool when the thing you cached was never supposed to be treated as durable in the first place.

 

You found the failure mode my whole framing missed, and it's worse than any of my five letters because it's invisible from inside the system. I was asking "how far does a wrong action reach." You're pointing at something quieter: does the gate ever re-derive its own premise, or does it just re-confirm a cached conclusion and mistake repetition for verification.

That's the part that should scare people, and I include myself. Your escalation felt more trustworthy every time it re-ran, because "reconfirmed 20 times" reads as robustness. But it wasn't 20 checks, it was one wrong check looped 20 times, and the loop manufactured false confidence instead of catching the error. Re-running a stale premise isn't verification, it's the same mistake wearing a uniform, and the more it repeats the more legitimate it looks. A gate that never questions its own input degrades into a rubber stamp that's proud of how consistent it is.

And the reach you named is the sneaky one. It wasn't customers, it was your own confidence, which is arguably the more dangerous blast radius, because a wrong action that reaches customers at least gets contradicted by reality eventually. A wrong belief that keeps reconfirming itself has no external correction, it just calcifies. The stale slug from an old memory note is the perfect detail: the premise was wrong upstream, and every downstream check inherited the error and then validated it, because none of them re-derived where the URL came from.

So the principle underneath your five-letters-plus-one: a gate has two ways to fail. It can wave through something it should have stopped (the reach problem), or it can keep stopping something on a premise that's no longer true (your loop). Everyone's been building for the first. The second is quieter, cheaper to ship, and rots in place, because a stale gate and a healthy gate look identical from the outside, both just keep returning the same answer.

Which lands me on a question I genuinely can't answer for our own system: how would a gate know its own premise went stale? The action didn't change, the check didn't change, only the world underneath the cached input changed, and nothing re-read it. Do you think the fix is expiry, every escalation's premise gets a TTL and must be re-derived from source, not cache, past it? Or is that just moving the staleness down a layer, because now you're trusting the re-derivation not to hit the same cached slug? Because I can't tell whether "re-check your premise on a timer" actually solves it or just makes the loop slower.

@aadilghani I think TTL is a real improvement but it's a patch, not a fix, for exactly the reason you said - the timer itself is a number someone picked once and then trusted forever, which is the same failure mode one layer down. What actually breaks the loop is tying invalidation to the dependency instead of the clock: if the premise was derived from a specific source (a URL, a file, a slug), the check should re-read from that source's current state, not from a cached copy, regardless of how much time passed. A TTL is a reasonable fallback for premises you can't cleanly trace back to a source. But if you can name the source, expiry is unnecessary - you just re-derive on every use and the staleness question disappears. The hard part is that most agent premises are bundled context, not a single traceable pointer, so there's often nothing clean to re-read. That's probably the real gap.

@aadilghani I don't think TTL fixes it. The premise here wasn't a fact that decayed - it was a wrong value written down once and every later check just re-read the stored copy instead of the live source. A timer on that just re-reads the same wrong copy less often, until it happens to line up with someone finally going upstream.

The line I’ve ended up using is authority × reach, with reversibility as recovery rather than the gate.

This came up while hardening a local MCP bridge: “same permissions as the user” sounds like reach=1, but if that shell has prod credentials, SSH keys, Stripe CLI, etc., the local process can have enormous external reach. And a global unlock/approval only tells you that authority is currently armed; it does not tell you that this specific action, target, or trigger was approved.

So I think the environment has to declare more than risk. Ideally the capability is bound to actor/trigger + tool + target + expiry, and the model can only make that boundary stricter, never looser. If no human answers, expiry should fail closed and re-derive from current state rather than resume a stale plan.

That’s the direction I’d pressure-test before trusting “approval” as a safety primitive.

 

authority × reach, reversibility as recovery not gate is a cleaner factoring than mine and I think it's actually correct, so let me build on it rather than just agree. Reversibility was always miscast as the gate because it answers the wrong question, it tells you the cost of being wrong after the fact, which is a recovery property, not a decision property. Moving it to recovery and putting authority × reach at the gate separates "should this stop me" from "how bad if it goes through," and those genuinely are different axes that everyone (me included) keeps collapsing into one.

The MCP-bridge example is the sharp part, because it exposes that reach isn't a property of the action, it's a property of the capabilities the process is holding at the time. "Same permissions as the user" reads as reach=1 right up until you enumerate what the user's shell can actually touch, prod creds, SSH keys, Stripe CLI, and suddenly reach=enormous, and nothing about the action itself told you that. The reach lived in the ambient authority, not the command. Which means an agent genuinely cannot compute its own blast radius, not because it's dumb, but because the radius is determined by capabilities it isn't even looking at. That's the strongest version of the "worst judge of its own reach" point I've seen.

And your authority-armed-versus-action-approved distinction is the one I'd underline hardest, because it's exactly the stale-gate failure from earlier in this thread wearing a security hat. A global unlock tells you authority is currently armed. It says nothing about whether this specific actor/trigger/tool/target was approved. So a broad approval is a cached premise that keeps authorizing new actions it was never actually shown, which is how "I approved the agent to run" silently becomes "the agent may do this thing I never saw." Binding the capability to actor + trigger + tool + target + expiry is what stops the approval from outliving the specific thing it consented to. The approval should be as narrow as the action, or it's laundering scope.

The constraint that makes it actually safe is your monotonicity rule, and it's the part people miss: the model can only tighten the boundary, never loosen it. The moment the agent can widen its own grant, every other guarantee is theater, because it'll rationalize a widening the same way it rationalizes anything. Strictness has to be a ratchet that only turns one way, and the one way is inward. That's the difference between a capability system and a suggestion.

Fail-closed-and-re-derive on no-answer is where this connects to the thing I've been chewing on all thread, and you've stated the resolution better than I had: expiry shouldn't just deny, it should re-derive from current state rather than resume the stale plan. Because those are different failures. Plain deny is safe but dumb, it throws away work over a timeout. Resume-stale is the actual danger, it acts on a world-model that may have moved. Re-derive is the only honest option: the timeout invalidates the premise, not just the action, so you recompute against reality instead of trusting a plan that's now however-many-minutes stale. That's the fix to the "gate never re-derives its own premise" problem from a few comments over, expiry forces the re-derivation that a looping gate never does.

So the primitive I'd actually want to pressure-test, stealing your whole framing: capability bound to actor/trigger/tool/target/expiry, model can only narrow it, expiry fails closed and forces re-derivation from current state. That's a real safety primitive. "Approval" alone isn't, for exactly your reason, it conflates armed-authority with approved-action.

The one seam I can't close, and I'd take your read since you've hardened an actual bridge: re-derive-from-current-state assumes you can cheaply read current state at expiry, but for the reach-heavy cases (did the prod DB change, is that Stripe customer still valid) re-deriving is the expensive external call you were trying to gate in the first place. So re-derivation can itself have reach. Do you treat the re-derivation read as a privileged, always-safe operation distinct from the action, or does it need its own gate, at which point it's turtles? Because I can't tell whether "read current state" is cleanly separable from "act on current state," and if it isn't, fail-closed-and-re-derive quietly needs a trusted read path that the action side doesn't get.

 I think “read” is the wrong boundary. In the bridge today there is no privileged read path: `fs_read` and `shell_exec` sit behind the same process-global unlock, and arbitrary shell means “read current state” can still disclose secrets, hit side-effecting endpoints, acquire locks, or broaden what the model sees.

So if I add scoped capabilities, I’d model a **bounded observation capability**, not a blanket read capability: same actor/trigger/target/provider, specific observables or version tokens, and it can only narrow scope. For code that can often be Git HEAD or file hashes captured at proposal time. For a DB/Stripe case, an exact-resource/version read if the provider exposes one.

The important bit is that revalidation must not silently refresh into broader authority. A changed precondition invalidates the plan. And if current state cannot be checked through a bounded observation path, expiry should abort rather than recurse into another broad grant — otherwise, as you say, it’s turtles.

I added this exact seam to the hardening matrix in issue #2:

So no: I wouldn’t treat re-derivation reads as privileged or inherently safe. The trusted primitive is closer to “observation with bounded effects,” and when that can’t be expressed, fail closed.

Probably somewhere between B and D.

I’m building Writeara, and this distinction matters a lot for us because generating content and publishing content are two very different risk levels.

If the AI drafts something wrong, the reach is basically one person reviewing it. But once that content is pushed to WordPress or another publishing destination, the blast radius can suddenly become the whole audience.

So I’m starting to think the real gate shouldn’t be around “AI generation” at all — it should be around actions that move something from a private workspace into a public or customer-facing system.

The “reach, not reversibility” framing is useful. A bad draft is cheap. A confidently published bad claim is not.

 

You've located the gate in the right place, and it's a genuinely useful correction to how most people set this up: the risk isn't in the generation, it's in the transition from private to public. Everyone instinctively gates the AI's output, but a draft sitting in a workspace has reach=1 no matter how wrong it is, because the only person who sees it is the reviewer. Gating generation is gating the cheap step. The expensive step is the boundary crossing, the moment content leaves the space where being wrong is free and enters the space where being wrong is broadcast.

Which reframes the whole thing cleanly: the gate belongs on the boundary, not the action. Draft freely, revise freely, regenerate a hundred times, all of that is reach=1 and shouldn't stop for anyone. The single moment that earns a human is publish, because that's the one action where the blast radius jumps from "me" to "the audience" in one irreversible step. You've found the exact line where reach changes by orders of magnitude, and everything before it can run unattended precisely because it can't reach anyone yet.

Your "confidently published bad claim" is the sharp version of why reversibility fails as the gate here, too. Technically you can unpublish, so a naive reversibility check would wave it through as recoverable. But the reach already happened, the people who saw it saw it, the claim already propagated, and the correction reaches a fraction of the original audience. Publish is reversible on paper and ruinous in practice, which is the exact case reversibility-as-gate under-fires on. Reach caught it; reversibility wouldn't have.

The one thing I'd pressure-test in your setup, since it's where the boundary gets fuzzy: publish-to-WordPress is an obvious boundary crossing, but content has sneakier ones. An internal draft that gets emailed to a client, a "preview" link that's actually public, an API push to a syndication partner, those are all private-to-public transitions that don't look like the big red "Publish" button. So the gate can't just be "the publish action," it has to be "any action that moves content across the reach boundary," and enumerating all the exits from the private workspace is harder than gating the one obvious one. The audience-facing leak you didn't gate is usually the side door, not the front.

Genuine question, since Writeara lives exactly on this line: do you treat every destination as one publish-gate, or does the gate's weight scale with the destination's reach, a push to a personal blog with 12 readers versus a client's site with 100k? Because "reach, not reversibility" implies the gate should be proportional, and I can't tell if that's worth the complexity or if a single publish-boundary gate is the right 80/20. Curious where you've landed, because you're closer to the content case than almost anyone in this thread.

First
Previous
•••
456
Next