How do you investigate AI agent failures after they happen?

Every production AI system eventually encounters failures. The challenge isn't just fixing them—it's understanding why they happened in the first place.

I'm interested in how teams investigate failures after deployment.

Do you collect execution traces, save intermediate reasoning steps, analyze tool calls, or build custom debugging dashboards?

What has made the biggest difference in helping your team identify the root cause of difficult AI agent failures?

I'd love to hear the workflows and tools that have worked best for you.