A simple AI governance checklist for your first production agent
A lot of agent projects spend months improving prompts, models, and workflows, then ship to production without asking a few fairly basic questions. Before trusting an AI agent with real work, I'd want clear answers to these:
1. What actions is the agent allowed to take?
Define the boundaries before deployment, not after.
2. Which tools and systems can it access?
Give access based on necessity, not convenience.
3. Are sensitive actions gated by approval?
Not every step needs review, but some decisions probably should.
4. Is there a record of what the agent did?
If something goes wrong, you should be able to reconstruct the sequence of events.
5. What happens when the agent is uncertain?
Low-confidence outputs need a clear path instead of silently moving forward.
6. Can you roll back mistakes?
Every production workflow should have a recovery plan.
7. Who owns the outcome?
At some point, a person needs responsibility for the workflow, even if an agent is doing most of the work.
None of these questions are particularly complicated. What's surprising is how often they only come up after an agent is already running in production.
What would you add to the list before trusting an agent in production?
Replies
One thing I’d add is monitoring for behavior drift. An agent may behave correctly during testing but start taking unexpected paths as tools, prompts, data, or surrounding systems change.
Exactly. Passing tests once does not mean the agent will remain safe as tools, prompts, and data change. Continuous monitoring is essential for catching that drift early.@ashir_murtaza1
I think production automation needs more than just prompt rules. Adding more rules can make an agent less flexible, but it still does not fully prevent hallucinations or unsafe actions.
A better approach might be to use specialized agents. Each one handles a specific task with full control, but only inside a limited context. That way, the system stays capable without giving one agent access to everything.
How do you enforce those limits during execution?
That is where runtime enforcement comes in. Limits should be checked outside the agent before every action, with scoped permissions, policy checks, and approval gates for sensitive operations.@flavio_riper
@grover___dev Yep, that is the boundary I would want too. The agent can propose the action, but it should not be the one deciding whether its own action is allowed. Do you also keep the exact policy version and evidence behind each check, so someone can see later why it passed?
I'd agree the exact policy version should stay in the record alongside the concrete evidence that check evaluated. Otherwise an audit log can show that an action passed without showing which rule or source made it pass. I’d also carry the intended outcome and inspect the result after execution: a perfectly authorized action can still be the wrong bet.
A most useful record is policy + evidence + decision + observed result, linked so a reviewer can reconstruct the path without trusting the agent’s summary.
Dial
solid list. I'd add one that's easy to skip if all your agent examples are text/API-based: does the action happen in the real world where it can't be undone. rollback plans work great for a database write or a sent email you can follow up on, but a phone call the agent already made and the person already heard has no rollback, whatever it said is now common knowledge between two humans. worth explicitly separating reversible actions from ones where the best you get is a fast correction afterward, not an undo
@omri_ben_shoham1 Strong point. Fast correction is useful, but it cannot reverse every action. The safest control is still preventing irreversible or high-risk actions before they execute.
good list. thing i would add is that every item on it is about before the action. nothing checks afterwards whether what claims to have happened actually did.
the failures i get are not blocked actions or errors. the reply sends fine, the trail says sent, nothing on the list is violated, and it was about the wrong order. the record tells you what was allowed and what ran, not whether it was right.
so 8 would be, how do you find out you were wrong when nothing errored. for us the honest answer is the customer tells us, which is not really an answer. anyone got better?
@jernej_jan_kocica Completely agree. Successful execution does not always mean the outcome was correct. Post-action verification is needed to confirm the intended result actually happened, rather than waiting for the customer to report it.
jernejs point about the post-action gap is the right one. adding a #4 that answers it: when an action is taken, whose name is attached as the responsible human, and can that person revoke consent later? most governance stacks stop at 'was it allowed' (pre-commit checklist) and 'did it happen' (audit log). the missing layer is 'is the human who approved it still standing behind it three months later.' consent-of-record instead of just reviewer-id. this matters most for actions that were technically-in-scope but produced outcomes the reviewer wouldnt sign for now. rare in aggregate, catastrophic when it happens.
@thenameisarian This is an important distinction. An audit log tells us what happened, but not necessarily who remains accountable for it. Linking sensitive actions to a responsible human and explicit consent would make governance much stronger.
Dial
Agreed on prevention over correction. The part I find genuinely hard in practice is drawing the line ahead of time - an email feels reversible right up until the recipient opens it before your follow-up lands, a database write feels reversible until something downstream already read the bad value and acted on it. Rollback plans tend to assume the world pauses while you fix things, and it usually doesn't. Curious if anyone's found a good way to classify 'reversible' that accounts for the time window, not just the action type.
Traceability is the only way to bridge action back to the evidence and intended outcome that justified it. The question I believe is still open is how to bring the result back to that same decision?
i.e. A clean execution log can still describe a perfectly authorized action that was the wrong bet. In product work, the failure often starts earlier: a customer signal gets summarized into a ticket, the reason thins out, and by delivery nobody can inspect what would count as proof. The audit trail needs both the control plane (scope, approval, action) and the learning loop (source, intended outcome, observed result). Otherwise post-action verification has nothing concrete to verify.
Curious how you’d represent that link without turning the checklist into a second bureaucracy.