How do you know when an AI agent is ready for production?

One question I've been thinking about is when an AI agent is actually "ready" for production.

Traditional software often has clear testing processes, but AI agents introduce a different level of uncertainty. They rely on reasoning, external tools, changing context, and non-deterministic outputs, making it difficult to define a clear launch criterion.

How does your team make that decision?

Is there a checklist, a target success rate, a set of benchmark tasks, or simply enough confidence after internal testing?

Have you ever deployed an agent that looked ready in development but behaved very differently with real users?

I'd love to hear how different teams decide when an AI agent is ready to move from testing into production.