Opt-in Self-Check
A runtime behavioral probing framework for LLM agents
4 followers
A runtime behavioral probing framework for LLM agents
4 followers
We built a framework that analyzes LLM agent behavior during execution instead of only final outcomes. It detects failure signals such as: - tool errors - repetitive actions - stagnation - degraded exploration This helps understand how and when LLM agents begin to fail during runtime, not just after completion.
Would love to see a way to set custom thresholds for each failure signal, since what counts as "stagnation" varies a lot depending on the task. Maybe even let teams define their own failure patterns on top of the built in ones.