Agent-Eval - Statistical regression testing for LLM agents
by•
Statistical regression testing for LLM agents. Run versions A and B 50 times to get a p-value, Cohen's d, and a 95% CI proving whether behavior actually shifted. While DeepEval, Braintrust, and Promptfoo test single responses against a threshold, none measure distribution drift. It is completely self-hostable under an Apache 2.0 license and requires no SaaS subscriptions. Works natively with LangGraph, OpenAI Agents SDK, CrewAI, and LangChain LCEL. Start testing: `pip install agent-regress-cli`.
Replies
The recurring failure mode is frustratingly common: an agent is updated, standard evals stay green, but performance noticeably degrades a week later. Two real-world examples catalyzed the development of this tool:
- LangGraph PR #4486: Added node-level result caching, which quietly masked the repeated-sampling variance a regression test depended on, something a plain threshold check has no way to catch.
- CrewAI PR #6134: Fixed a bug where file tools leaked absolute filesystem paths in responses. A standard quality scorer never flags this because it only checks if the answer is correct, not what is hidden in the payload.
These examples are part of a larger validation pass encompassing 29 PRs and 239 rows across LangGraph, CrewAI, and the OpenAI Agents SDK. The full data breakdown is available in the pr-analysis.md documentation on github.
Instead of grading a single response, agent-eval statistically compares two whole batches of runs. By running versions A and B 50 times each, the tool executes:
- A Mann-Whitney U test (requiring no assumption that scores are normally distributed).
- A bootstrap confidence interval to confirm if the shift is real.
- Cohen's d to determine if the shift is large enough to actually matter.
A p-value under 0.05 by itself is not enough to call it a regression. The effect size is required for definitive proof.
As part of statistical guardrails, two limits are utilized to ensure data integrity.
- Under 30 runs per version: Triggers a warning due to a drop in statistical power.
- Under 10 runs: Returns INSUFFICIENT_DATA rather than guessing at an answer.
This is one piece of a larger observability, testing, and benchmarking infrastructure designed for production agents. We are already planning for the next version.
Repo: https://github.com/RudrenduPaul/agent-eval
MCP Servers:
https://mcpservers.org/servers/rudrendupaul/agent-eval
https://glama.ai/mcp/servers/RudrenduPaul/agent-eval
NPM: https://www.npmjs.com/package/agent-regress-cli
PyPI: https://pypi.org/project/agent-regress-cli
dev.to article: https://dev.to/rudrendu_paul/threshold-testing-is-blind-to-agent-regression-we-built-the-statistical-alternative-validated-21lh
Should AutoGen or the Vercel AI SDK be prioritized next? What do you suggest?