Your agent does something wrong. How far does it reach before anyone notices?
Two months ago I asked here how people stay aware of what their agents are doing, and @getosmo replied with something that has been quietly rearranging my roadmap since.
I had been treating "can you undo it" as the line for when an agent should stop and ask. He took that apart. Irreversibility over-fires and under-fires, he said. It over-fires on things that cannot be undone but that nobody cares about, like the agent writing a log line. It under-fires on the ones that really hurt: a mass email to 50k users, or a migration you can only roll back after downtime. "Reversible on paper, ruinous in practice."
His version is that the gate should key on reach, not on reversibility. Not "is this safe" but "how far does this reach if I'm wrong." And the part that stuck with me: the agent is the worst possible judge of its own blast radius, because it does not know there are 50k people on that list.
@tina_chhabra had pushed the same way earlier in that thread, that this gets worse the moment the agent stops writing code and starts booking meetings or sending email, where you find out after the fact. I have been sitting with both of those for two months, and they leave me with a question I still cannot answer cleanly for our own product:
If your agent gets one thing wrong today, what is the largest number of people who find out before you do?
For a coding agent that is usually one. You, reading a diff. The moment the agent sends, books, charges, replies to a customer or touches production, that number stops being one, and most of us have never actually worked out what it is. So whether you run agents or ship them to other people, which of these are you?
A. Nothing my agent does reaches past me. Genuinely low stakes, no gate needed.
B. It could reach customers, I know it, and I have not built the gate yet.
C. It stops and waits for someone to come back and look. Safe, and it quietly kills completion.
D. It emails or Slacks someone, which mostly works until the person who cares is not in that channel.
E. I built the real thing, timeouts, escalation, an audit trail, and it ate a quarter I had not planned for.
Ours, honestly: we sat on B longer than is comfortable, moved to D, and then found that delivering the question is the small part. What surrounds it is the work. Whether a timeout should mean deny or hand back to the terminal took us longer to settle than building the delivery did. So did reaching the one person who can answer
when they are not sitting in front of your product, and being able to prove afterwards who approved what.
The two I still cannot answer, and I do not think anyone has:
How do you reach the person who has to decide, when they are not looking at your product and never installed anything of yours?
And what does your system do when they simply never answer? Every answer I have heard to that one is a guess, and the guess is usually "keep waiting", which is how a job sits stalled for six hours and a customer decides your product is broken.
[Same disclosure as last time: I build Pushary, which is the ask-a-human part of this, and these days we sell it to other companies to embed as much as to people running agents in their own terminal. Obvious bias, asking anyway. The last thread taught me more about this than six months of building did.]
Which letter are you, and what did it cost you to find out? I will come back in a few weeks and post the spread.
Replies
You are probably right that the near miss usually shows up as an annoyed human, but the version that keeps me up is the one where nobody is annoyed at all. The dangerous output is not the one that looks broken, it is the one that looks fine. Plausible and wrong generates no complaint, so the detection channel everyone here is describing, the irritated customer, only fires on the failures that happen to be visible. The quietly consistent ones sail straight through, because the absence of a complaint reads exactly like success. So the honest answer to your question is that the largest number who find out before I do is not the scary number. The scary number is the people who find out and never say anything, because nothing looked wrong enough to mention. On the letters, we are a D that keeps catching itself having been a C on paper, the same trap Zeeshan described. And Clemente's point about acknowledgement being a separate recorded event from the decision is the piece I would build first, partly because it also chips at this: if seeing the action is its own event, you at least know a human's eyes crossed it, even on the days nobody complained. Great thread.
@asadmalik901
The inversion is the sharpest thing in this thread and I think it breaks my own question. I asked how many people find out before you do, which quietly assumes finding out is an event. Your version is that the number that matters is the people who found out and said nothing, and that number is not observable at all. So the metric I opened with only ever measures the failures loud enough to be measured.
The thing that ties your point to Clemente's, for me, is that we are using silence as evidence in two opposite directions at once. In the approval flow, no answer means we do not know what they thought. In the outcome, no complaint means it worked. Same absence, opposite conclusions, both guesses.
Which is why your extension of his idea is better than the original. In a system where failure is quiet, acknowledgement is the only positive signal available. Everything else is measuring an absence. Ack turns "nothing happened" into "someone looked and nothing happened", and those two states are identical today despite being nothing alike.
One caveat, since otherwise this flatters my side of it too much: a tired person will ack a plausible, confidently wrong reply and approve it. Ack proves it was looked at, not that it was seen. It moves the failure from undetected to detected and waved through. Progress, not a fix.
The only traces I can think of for the silent version are second order. The customer who does not complain but asks the same thing again a week later, or quietly stops replying. Have you found any of those that actually hold up, or is it all after the fact?
we build voice agents, so our version of this is a bit different from the send-an-email case. the gate on whether the agent CAN place a call or send an SMS is easy, that part is basically E already, permissioned, logged, rate limited. the part nobody has a clean answer for is what happens inside a call that's already live. the agent can say something that's technically within its instructions but just not what a human rep would have chosen to say in that moment, and by the time anyone reviews the transcript the customer already heard it and reacted to it. there's no pause-and-ask step mid sentence. so our real answer is closer to D, we catch it after the fact through review and adjust the prompt, and we're honest that this is weaker than we'd like for anything customer facing in real time.
@omri_ben_shoham1
This is the case that breaks the assumption underneath the whole thread. Everything above quietly assumes the action has a before. A send has one. A diff has one. A live call does not, because the unit of action is the entire conversation and it is already in flight.
Which makes me think approval is the wrong concept for you rather than a missing feature. The only mid-flight intervention that exists in voice is handoff, getting a human onto the line. So the question stops being can the agent pause and ask, and becomes how fast can someone join and what makes them join. That is a latency problem, and latency problems are solvable in a way that pause-and-ask is not.
The thing you have that nobody else in this thread does: your damage signal arrives while you can still act on it. Jernej made the point further up that detection and damage are usually the same event, which is why you always find out too late. In voice they are still the same event, but it is continuous. The customer interrupting, repeating themselves, going quiet, all of that lands while the call is open. Everyone else here finds out tomorrow. You find out in four seconds.
Straight about my own side: for the mid-call part, Pushary is not the tool. It is a pre-commit thing and you have no pre-commit moment. Where it would apply is the handoff, reaching whoever can actually take the line when they are not sitting in your dashboard, which is the same reachability problem one domain over.
So, do you have anything today that can pull a human onto a live call, or is the honest answer that the call just finishes?
Abdullah's reply upthread is the one I'd put money on being the real gap, and it went quiet, so — same experience here, from the other side of it.
I run routines that push a few dozen outbound items a day, every one gated by me before it goes. The gate was never the weak part. The completion message was. The tool reports success because from where it sits the click landed, and what's actually on the page is truncated, or posted into the wrong composer, or went out twice. So the last step of every routine is now: reload the rendered page, find the item, confirm it exists and says what I approved, and only then write it to the log. That check has caught more real failures than the approval gate has, and it costs seconds.
Which suggests a third layer next to Tyler's two. Reach is prevention, reversibility is recovery, and neither asks whether the thing you approved became the thing that exists. An action that silently didn't happen and one that happened differently both report identically.
The other thing I'd add to your letters is rate. Every one of them describes a single action. The failure I actually design against isn't a bad item, it's a run that emits fifty individually-fine ones inside an hour — each clears a per-action gate, because each is genuinely fine, and the damage lives entirely in the count and the clock. It's also where batch approval quietly becomes the problem: approving a list of fifty is the point at which reading them stops. Reach asks how far one action travels. Nothing in the framing asks how many times.
@narek_keshishyan emdashes here are wild
@aadilghani Ha, fair. Yeah, I use Claude Code for most of what I build, and these go through it. The drafts are mine though: my thinking, my call, and nothing goes out that I haven't read line by line.
Which is on topic, I suppose. That's the gate I was describing, and you just found the tell it doesn't catch. Taking the note on the dashes.
@aadilghani Ha, fair pitch. Honestly the gate is the part I already have. What I don't have is the half your own thread is about: an approve on the lock screen is a yes to an intent, and the thing that actually burns me is the gap between the intent and what landed. I approved the right action, the action half-worked, and I found out later from a person.
So the push I'd pay for isn't "may I", it's "here's what actually exists now" once the run is done, the rendered result rather than the agent's word for it. Approvals I can batch. Verification is what I keep having to do by hand.
honest answer today: the call just finishes. we can page a human mid-call, an alert goes out the moment something looks off, but paged and actually bridged onto a live line before the caller hangs up are very different timelines. getting someone from notified to present in under the window a caller will tolerate is the part we have not solved, so right now the realistic outcome is a human reviews the transcript afterward and calls the person back, not a live rescue. agree it is a latency problem, not a permissions one, that is the useful reframe here
The person who never installed anything is a paging problem, not a notification problem. On-call escalation policies exist because you can't assume the primary is watching any single channel: push, then SMS, then a phone call, escalating on a timer until someone acks. What's different here is you can't push into an app they don't have, so the first hop has to be a link, not a notification. That link is now a bearer token to a real decision. What stops it from being forwarded, screenshotted, or clicked by someone else in the thread who isn't actually the decider?
@mohsen_bashirzadeh
Paging is the right frame, and yours is the first comment here that names the thing our routing was actually built for.
You are right that the link is a bearer token, and by default that is all it is. Ours is HMAC signed and resolves once, first answer wins, atomically. That covers forgery, enumeration, and replay after the fact. It does nothing about the case you are describing, which is the wrong person inside the window. Signing proves the link is real. It does not prove the holder is the decider.
What does cover it is an enforced route: the question carries the set of users allowed to resolve it, and a forwarded link does not work when someone outside that set clicks it. Two details a paging person will care about. The route is frozen onto the question at ask time rather than resolved at answer time, so editing the on-call rule mid flight cannot retroactively change who was allowed to answer, which would otherwise quietly rewrite the audit trail. And a broadcast surface that cannot say who acted is excluded by construction, so your shared thread case falls out of the same rule instead of needing its own patch.
Then the honest part. Enforcement requires the decider to be a named user on our side, and the person in my original question is exactly the one who is not. So the control exists and it does not cover the case that motivated the post. Identity has to be established before the decision, never at it. Either you have a prior relationship you can enforce against, or you have possession of a link, and there is no third thing.
Which is also why my two open questions turn out to be one. Add a second escalation hop and first click wins stops being a policy and becomes a race. You cannot escalate to a second person if you cannot tell them apart from the first.
Where does your ladder get the identity from, before the first page goes out?
@aadilghani Nowhere good, and that's the honest answer. Paging systems don't solve this, they dodge it. An escalation ladder works because everyone who could ever get paged is already a named identity in the company directory before the first incident happens. The pool is closed and provisioned in advance. That's still your first bucket, prior relationship, just pushed back to onboarding instead of ask time.
Your case breaks the analogy exactly where you're pointing at it. The decider was never added to any directory of yours. There's no pool to provision them into ahead of time, because you don't know who they are until the moment you need them.
So paging isn't a fix, it's a description of the easy case. The hard one, a genuinely external decider, is the one you already named: possession of a link, or nothing.
@mohsen_bashirzadeh
That is a better description of paging than most paging docs manage. The pool is closed and provisioned before the incident, so identity is never a decision-time problem. It is an onboarding problem that already happened.
One place I would stretch your binary. The closed pool does not have to be mine. If the decider already belongs to someone else's directory I can borrow it, so an answer arriving through Slack carries a Slack user id and gets recorded as a namespaced external identity rather than as one of my users. Still your first bucket, just not my onboarding. That covers more real cases than it sounds like, because most external deciders are external to me and internal to something.
When neither exists, agreed, it is possession or nothing. Which is why the part I care about moved off decision time and onto the record.
Three actor kinds, deliberately not collapsed. A named user. A namespaced identity borrowed from another system. And anonymous, meaning the answer arrived over a credential that carries no identity at all. A push to a device belonging to one of my users mints the link with a second signed binding to that recipient, domain separated from the answer signature so a one-tap approve link cannot be upgraded into a claim of identity. The device class in your scenario, an end user of a partner who has no account with me, gets the URL unchanged, because there is nothing to bind it to. Not an oversight. The same fact showing up one layer down.
Why that matters more than it looks. A wrong approval is a failure you might catch, because something downstream breaks and you go looking. A record that reads as though it knew who acted is a failure you will not catch, because nothing looks wrong.
The cost, honestly. That path degrades to anonymous rather than refusing, so a stripped or tampered binding loses the attribution and keeps the approval. Availability over attribution, on purpose, because a decision nobody can resolve blocks the agent. Same trade a pager makes when it wakes the whole rotation rather than risk waking nobody.
In your ladders, have you ever had an ack that could not say who acked, and did anyone treat that as an incident?
@aadilghani Yes, and it never got treated as an incident when it happened, only afterward.
Shift handoff is where it happens. Two people are both technically holding on-call credentials for the ten minute overlap window, and an ack that lands inside that window comes from "on-call," not from either of them specifically. Nobody notices at 2am, because the alert cleared and the system moved on. It only becomes visible in postmortem, when someone asks who actually looked at this and the honest answer is the log says on-call acked it, not a name.
That is your own point arriving from the other direction. The failure was never in the ack. It was in the record believing the ack meant more than it did.
We ended up doing the version of what you're describing without having a name for it. The handoff window gets logged as its own state, so an ack inside it is stamped ambiguous rather than attributed, instead of silently inheriting whichever person's shift technically started. Same trade you named. We'd rather the record admit it doesn't know than guess and be wrong quietly.
I would make the default decision proportional to both reach and recoverability, but keep the policy outside the agent. The agent should receive an action budget: which boundary it may cross, the maximum audience or spend, and what to do on timeout. That turns a vague “ask when unsure” rule into something testable. The important missing piece is a safe fallback: a timeout should not simply pause forever or silently approve; it should either narrow the action (for example, create a draft instead of sending) or fail closed with enough context for the next human to resolve it quickly.
@ariajinling
Outside the agent is the part I would not compromise on, and the reason is stronger than testability. A policy the agent evaluates is a policy the agent can be argued out of, because it inherits the same context and the same pressure that produced the questionable action in the first place. A line from earlier in this thread applies: you cannot build a tripwire out of the thing that is tripping. Testable is the benefit you named. Non negotiable is the one that carries it.
On the fallback you have named the piece almost everyone skips, and I want to point at the condition it rests on. Narrowing works when the action has a weaker form that already exists. Send has draft. Publish has staging. Charge has authorize without capture. Plenty of actions have no proper subset: a refund has no half refund, a schema migration has no rehearsal that is also progress. Where no weaker form exists, narrow quietly collapses back into fail closed, and you have a ladder with one rung wearing a costume.
Which argues for making the fallback a declared field on the rule rather than something resolved at timeout. If you cannot name the degraded form while writing the policy, there is no degradation available later. Writing it down at authoring time is also how you find out how few of your actions have one.
There are two levers here and they are orthogonal. Blake in this thread argued the timeout should widen the ask, next person, different channel, before it means anything final. You are arguing it should narrow the action. Both run on the same clock and neither substitutes for the other. Most systems ship one and treat the problem as solved.
Where I should be straight about ours. Our timeout answers are approve, deny, or escalate to someone else. All three are decisions about the action. Degrade is a fourth answer that returns a modified action, and our protocol cannot express it. That is a shape gap rather than a missing feature, and it is the more useful thing you have told me.
Your second half is the underrated one. Fail closed without context only relocates the stall. And the context that matters is not the action being blocked, it is what already completed before the stop, because partial work is what makes the next human slow.
So, of the actions you would actually put a budget on, what fraction have a real weaker form?
I agree: degradation has to be an explicit, author-authored alternative, not a timeout-time improvisation. If an action has no degraded form, it needs a stronger escalation path or a shorter approval window. For partial work, I would preserve a compact execution ledger: completed steps, external effects, and the invariants re-checked. The human then decides from the delta: what was attempted, what changed, and what remains safe to do.
The shift from “reversible” to “blast radius” is a really important distinction. The real challenge with agents isn’t just knowing when to ask for approval, but knowing who to ask, how fast, and what happens when no one responds. That’s where human-in-the-loop becomes a systems problem, not just a UX feature.
@kailong_20
Systems problem is the right frame, and it is worth naming why, because the three questions you listed are not three.
Who to ask, how fast, and what happens when nobody answers. The third one is not independent. It is what the first two look like when they are unresolved. If you can name the decider, no response becomes this specific person did not respond within N, which is something you can act on. If you cannot name them, every timeout is indistinguishable from a wrong address, and a slow human looks exactly like someone who never received it at all. You cannot build a timeout policy on top of an unresolved routing question, because you do not know what the timeout is measuring.
The failure modes do have names already, and the on call paging comparison came up further up this thread. The point where the borrowed solution stops transferring is that paging assumes the responder has your tooling installed. Here the person who has to decide often does not, so the first hop has to be a link rather than a notification, and a link is a bearer token for a real decision. That one property breaks most of what escalation policy design quietly assumes.
The place I would push back is on not just a UX feature, because it undersells the half that stays UX and stays load bearing. Routing decides whether the right person is reachable. What the ask actually says decides whether the answer they give is any good. Route perfectly to the correct human, who then approves a large charge because the request rendered as run a command, and you have solved the systems problem and still lost the money. Reach has to travel in the payload, not only in the policy.
So the thing that determines which of your three is actually hard for you: is the person deciding someone who already has your software, or someone who does not?
@aadilghani The harder case is definitely when the decider doesn’t have the software. At that point, the problem isn’t just routing. It’s context delivery. The human needs enough information to make a good decision without needing to understand the system behind it. That’s where the real complexity starts.
@aadilghani the reach > reversibility framing lands for me alongside a second failure mode: context loss
In our own workflow, the problem wasn’t only that an agent could take a risky action; it was that the reviewer could no longer see why the action existed, which customer signal it came from, or what evidence would prove it worked.
We’ve been treating the action, source, and intended outcome as part of the handoff, then bringing the shipped result back as evidence.
That makes me a cautious B: human approval is still needed when work crosses into customers, but approval only helps when the person can inspect the reason and provenance.
As in: faster execution makes missing context more costly, not less.
@jalcantara
Context loss is the right second axis, and there is a sharper version of it hiding inside your own description.
You listed action, source, and intended outcome. Those three do not have the same trust properties. The action is observable, the system records it whether or not the agent cooperates. The reason is narrated by the agent, which means you are asking the thing whose judgment you are checking to also supply the justification for its own behaviour. A model that decided to send is a model that can write a persuasive account of why sending was correct. So the reason field fails in the one direction that matters, and it fails invisibly.
That is the line I would draw, captured provenance against narrated provenance. The message id, the row the query returned, the retrieval that actually happened, those are captured and they are checkable. A paragraph explaining the motivation is narrated. And narrated provenance is arguably worse than none, because a reviewer reading a plausible reason approves faster than one reading nothing. It manufactures confidence rather than supplying it.
Your closing line has a mechanism under it worth spelling out. Review time per item is roughly fixed, bounded by human attention. Production time has gone to nearly zero. So the ratio inverts and the bottleneck moves entirely into reconstruction. Once reconstructing why takes longer than producing the thing took, people stop reconstructing and start checking whether it looks right. Nobody decides to make that switch, it just happens. Surface plausibility is exactly the test a fluent narrated reason passes.
The return leg does something I had not given it credit for. If you declare an intended outcome at ask time and then cannot produce a matching observation afterwards, the outcome was decorative. So the evidence coming back grades your handoff format, not only the work. That is a free test of the thing that is hardest to evaluate directly.
Where ours falls short of your bar: our ask carries the action, the target, and free text. There is no structured field for a source, so provenance in our asks is whatever the agent chose to say about itself, which is precisely the narrated kind I just argued against. A field that resolves to an artifact instead of prose is the fix, and we do not have it.
So on yours: does source resolve to something a reviewer can open, or is it a sentence?
It needs to resolve to something the reviewer can open, as a sentence would simplify too much the original intent. In the Zentrik model for instance, the original signal (call, ticket, research item, or experiment) stays linked to the decision and work it informed.
On the return leg shipped result and customer response should be inspectable against the intended outcome, not another agent-written summary. The practical test is whether a reviewer can reconstruct why the bet existed without trusting the agent to explain itself.
That’s about the bar I’d want a structured source field to meet.
@jalcantara
A reference, then, not a field. And the property doing the work is not that it is structured, it is that the referent exists in a system the agent did not author. A string can be written. A ticket cannot be made to have existed.
That puts the check earlier than it looks. If the reference only resolves when a reviewer clicks it, a well-formed identifier passes, and a reviewer under time pressure treats the presence of a link as evidence of a referent. Same failure as a narrated reason, with better formatting. To meet your bar it has to be dereferenced at ask time against the source system, and the ask has to fail when it does not resolve.
The return leg then constrains the intent field harder than I had it. If the shipped result is inspectable against the intended outcome, the intended outcome has to be written in the vocabulary the observation will arrive in. Otherwise nobody can compare them, a human writes a summary of the gap, and narration is back one level up. So the authoring-time test is whether you can name the thing that will come back. If you cannot, the outcome field was decorative before anything shipped.
Where that lands on me. We do carry an intent line. Optional, 500 characters, written by the agent, and the tool schema describes it as the user's stated task from their last prompt. Narrated twice, and exactly the sentence you just ruled out. The structural reason is that we ask from inside a tool call, where the agent is the only thing in the room that knows why. Provenance cannot originate there. It has to be attached upstream when the work is dispatched and carried through, which makes an approval layer the wrong place to author a source and the right place to refuse to lose one.
One thing I would want from Zentrik. What happens to an action with no upstream signal. Optional, and the field is empty exactly where scrutiny matters most, because unsourced work is the risky kind. Mandatory, and the agent finds something to point at, so narration returns as selection instead of fabrication. Which way did you go?
@aadilghani I wouldn’t necessarily make provenance mandatory in a way that allows the agent find a plausible reference. i.e. that can become a solution in search of a problem.
I’d keep the work explicitly unsourced, surface that gap, and require a human to supply or confirm the reason and intended outcome before treating it as an evidence-backed bet. When a real call, ticket, research item, or experiment exists, that artifact should stay linked upstream. If it doesn’t, the missing signal should lower confidence and change the review path... not get papered over by a narrated justification.
A, honestly, and I'm not sure that's a badge of honor, it's mostly that my agents don't have the surface area yet, not that I engineered it that way.
Everything I run agents on is coding-related, side hustle SEO tools, so worst case is a bad diff I catch on review. No customer sends, no charges, nothing touching production without me in the loop first. So "reach" for me caps out at one person: me, reading output before it ships.
But your framing makes me suspicious of my own answer. "Nothing reaches past me" is true right up until you automate something you didn't think of as customer-facing, a slug rename, a sitemap regen, a scheduled content push — and it turns out Google or your users were the blast radius the whole time, you just didn't have a human in that chain to notice.
Reach isn't just who your agent messages, it's who's downstream of the thing it touched. I'd bet a chunk of the A's in your replies are actually B's that haven't been wrong yet.
The timeout question is the one I'd want your data on more than the philosophy. "Keep waiting" isn't neutral, it's a silent default toward denial that costs you a stalled job either way, so why not make the timeout itself the escalation trigger? No answer in N minutes doesn't mean deny, it means widen the ask (next person on the list, different channel) before it means anything final. The person who never installed your product and never answers isn't a policy problem, it's a routing problem, you need a chain, not a single ask.
What's your actual N right now, and did you land on widen-then-deny or something else?
@blake_copeland
Numbers first, since you asked for them.
The default ask window is 60 seconds for most tools and 45 for edits. Read only work is set to zero and never asks. Destructive actions are configured so that no timeout can auto anything, they hold the full window and then ask. The phone gets the first stretch of that window and the terminal gets the remainder.
The part that surprised me when we measured it is that N is not really ours to choose. The agent harness kills the hook on its own budget, and those budgets differ by agent. Claude Code allows about two minutes, Cursor allows one. We take that number, subtract a guard so we can return a decision rather than get killed mid wait, and that is the ceiling. Any policy timeout longer than the harness budget is fiction. So the honest answer to what is your N is that it is bounded by somebody else's timeout and the policy only picks a value inside it.
Now the part where you are right and I have to concede it.
We escalate across channels, not across people. A timeout on the phone falls back to the terminal, and the terminal is where the agent asks its own question. That works because there is usually a human sitting there. But that is the easy case, and it is not the case my post was about. For the person who never installed anything, falling back to the terminal is falling back to a room they are not in. We can name a specific decider and enforce that only they may answer. We cannot sequence two of them on a timer. The chain you are describing does not exist in our product, and calling it a routing problem rather than a policy problem is the correct diagnosis.
Keep waiting is a silent default toward denial is also right, and it is worse than it sounds, because that denial is unattributed. Nobody decided it, so nobody is accountable for it, and it does not appear in any record as a decision.
On your own answer, the slug rename is a stronger example than you gave it credit for. A send has reach you can count at the moment you send it. A slug rename does not. Its blast radius is inbound links and an index, it produces no error, and it lands over days. So there are two kinds of action here: ones whose reach is a number at the time of acting, and ones whose reach only becomes a number later. Everything on the audience or spend axis is the first kind. Almost everything in SEO is the second. Thresholds and budgets only work on the first kind, which is why your domain feels like it has no natural gate.
The line I would keep is yours. A is a claim about your current wiring, not about your agent. Wiring drifts and nothing fires when it stops being true, so A is accurate as of the last time you looked and there is no event that tells you it expired. Which is why I would take that bet.
D, and I only got there by way of B.
The framing I landed on wasn't irreversibility either, it was blast radius plus time to notice. I run a support triage agent and an invoicing tool that can email clients. Both are reversible on paper. An invoice can be voided, an email can be followed with an apology. Neither of those matters, because the damage lands the moment another human reads it. I can't unsend it out of someone's head.
So the two things I gate on are how many people outside my org see this, and how long before I find out it happened. High on both, it stops. A log line, a file write, a row in a table I own, all of that runs free whether it's undoable or not.
What it cost me to find out: nothing dramatic, just a stretch of nobody looking. The agents had stopped running entirely and nothing said so, because the design treats no output as normal. Most of my incidents haven't been the agent being wrong, they've been the gap between wrong and noticed. That gap is the part I underbuilt.
On your unanswerable one, the person who never answers, I don't have a good answer either. Mine defaults to keep waiting, which is exactly the stall you describe. What I'd want is a deadline on the ask itself. No answer in N minutes means fall back to the safe option and say so loudly, rather than hold the line forever. A stalled job that took the cautious path is recoverable. A stalled job that just sat there is a support ticket.
@dalemooney
The two things you gate on are not the same kind of thing, and separating them changes what can fix each.
Blast radius is a predicate on the action. You can evaluate it before anything happens, which is what makes it gateable. Time to notice is an outcome. It is not knowable at the moment you would need it, only afterwards, and it is a property of your organisation rather than of the call. So it cannot be an input to an approval decision at all. It can only change how much you let run between checks. One axis is a gate, the other is a leash length, and treating the second like the first is how it stays underbuilt.
On the org boundary you have the cleaner version of something I have been circling in another branch here. Reversible means reversible by you, and your undo authority stops exactly where the last copy you control does. Voiding the invoice reverses your ledger. It does not reach the client's inbox, their forward to their accountant, or their opinion about your billing. So the question is not whether the action has an inverse, it is who else is holding a copy by the time you want one.
The three days is the part I would sit with, and it is a limit on what I sell. A gate would not have helped. Nothing asked. The agent was doing things that were individually permitted, correctly permitted, and wrong in aggregate, and an approval layer tuned perfectly still produces your Friday. Approval is a control on discrete acts. Drift is about aggregates, and there is no per call decision that catches it.
What catches it is a system whose silence is distinguishable from health, which is also what makes your deadline fix work. Say so loudly is the load bearing clause, not the fallback. A quiet safe default is a small version of Tuesday to Friday, a decision the clock made that nobody knows about.
Ours is honest about half of that. A timeout is a first class outcome in our records rather than a missing answer, so the clock's decision is written down as itself. But the record is pull, nothing pushes it. And the only periodic report is weekly and fires only for workspaces that had activity, so it cannot report silence and cannot resolve a Tuesday to Friday inside one window.
What I would want to know from your Friday: was there a signal present that nobody was watching, or was there no signal at all? Those are different builds.
@aadilghani The gate versus leash distinction is right, and sharper than what I wrote. Blast radius is knowable before the call. Time to notice isn't knowable at that moment at all, so using it as an approval input was me collapsing two different things. Leash length is the better name for what I was describing.
To your question, and it's the less flattering of the two answers: there was no signal at all. Worse, the absence of one was the designed success case.
The agents run on a schedule against a support inbox. There's a precheck that exits early when there's nothing to process, and an early exit writes no log line, because at the time not recording that nothing happened seemed obviously correct. So a quiet day and a dead agent produce identical output, which is none. I built the ambiguity myself and then read it as reassurance.
What actually broke was an auth token lapsing. Every agent that depended on it died at once. What hid it was that the other services on the same machine were unaffected, so anything asking is this host healthy answered yes, correctly, the whole time.
Which is why silence distinguishable from health is the right frame, and I'd push it further for your build. Not alert on error. Let me distinguish ran and had nothing to do from did not run. If skipped is a first class emitted outcome rather than something inferred from a gap, silence becomes meaningful and you can alarm on it. Your timeout being first class in the record is that same instinct, it just needs the push half. And a periodic report that only fires for workspaces with activity is structurally incapable of reporting the thing that matters most, which is a workspace that has gone quiet.
@dalemooney
You did not have a monitoring gap, you had an encoding one, and that is the harder kind because it sits upstream of anything you could have watched. Nothing to do and did not run produced the same output, so looking more closely was never going to help. A distinction the emitter dropped cannot be reconstructed downstream.
What makes that trap so reliable is that the collapse is both the cheap option and the locally correct one. Not writing a line when nothing happened is what a careful engineer does. Noise reduction and silence detection pull against each other, and the record that costs the most to keep is the one saying nothing happened.
The health check answering yes correctly is the sharpest part of it. Monitors get placed on ownership boundaries, hosts and services, things that have an owner. Failures propagate along dependency edges. Your token's blast radius was every process holding it, which was not a unit anything watched, and it cut across the machine the checks were organised around. A true answer to a question nobody needed answered.
Taking your design point, with one addition. Emitting skipped only buys you an alarm if something declares when it should have arrived. Your schedule supplies that for free. So the primitive has two halves, the emitted non-event and the expected cadence, and the second is the one we lack, because our agents are not scheduled. A human starts them, so a quiet workspace is usually a person on holiday, and alarming on that floods everyone.
Inside a session it is unambiguous, and we already carry the seam. Sessions have a last seen time and a sweeper walks them active, idle, offline as it goes stale. Then your comment sent me to look at what offline actually means, and we have built your exact collapse. A clean exit writes offline. A session that simply stopped reporting gets swept to offline. Same value, same column, and the board files offline under done. An agent that died and an agent that finished land in the same bucket, and the bucket is named after the success case. There is a comment in that file explaining the choice and it reads as obviously correct.
One thing I would check on yours before leaning on skipped. Make sure it means I looked and there was nothing, not I could not look. An empty inbox and an unreachable inbox both return zero items, and a precheck is exactly where that ambiguity gets rebuilt one level down.
@aadilghani Encoding rather than monitoring is the right correction and I am taking it. The precheck point lands too. "I looked and there was nothing" and "I could not look" both return zero, and a precheck is exactly where I would have rebuilt the ambiguity one level down without noticing.
What I take from that is that it is not a bug you fix once. Every layer that can report nothing gives "could not look" a new place to hide inside "nothing", and at each site the collapse is the locally correct choice. So vigilance is the wrong defence, because it asks someone to make the globally right call at the exact moment the locally right one is obvious. The only thing that holds is making the collapse inexpressible. If the only way to say nothing is a value that carries why it was nothing, the careful engineer cannot accidentally write the ambiguous version, because there is not one to write.
On your offline finding I would push slightly, because renaming the bucket does not fix it. The test is not whether the two states are distinguishable in the column. It is whether anything downstream branches on the difference. If the board does the same thing either way, splitting the enum relocates the collapse into whoever reads it. So the question before the schema change is what the board should do for stopped reporting that it does not do for finished, and if the honest answer is nothing, the split is cosmetic and nobody will notice when it regresses.
On cadence I think you have the half you say you are missing, you are looking for it in the wrong place. A schedule is a promise made in advance, which is why it comes free. A human started session makes its promise at start time: somebody deliberately began something, and that is a commitment that it will either progress or end. So the expectation is not "this workspace should be busy", which is unbounded and fires on holidays. It is "this specific started thing should not go quiet without finishing", which is bounded, and cannot fire when nobody started anything. Your last seen sweeper is already the mechanism. It is sweeping into the wrong bucket rather than lacking the signal.