Agent-Eval
Statistical regression testing for LLM agents
Start new thread

Agent-Eval - Statistical regression testing for LLM agents

by•
Statistical regression testing for LLM agents. Run versions A and B 50 times to get a p-value, Cohen's d, and a 95% CI proving whether behavior actually shifted. While DeepEval, Braintrust, and Promptfoo test single responses against a threshold, none measure distribution drift. It is completely self-hostable under an Apache 2.0 license and requires no SaaS subscriptions. Works natively with LangGraph, OpenAI Agents SDK, CrewAI, and LangChain LCEL. Start testing: `pip install agent-regress-cli`.

Add a comment

Replies

Best
Maker
📌
The recurring failure mode is frustratingly common: an agent is updated, standard evals stay green, but performance noticeably degrades a week later. Two real-world examples catalyzed the development of this tool: - LangGraph PR #4486: Added node-level result caching, which quietly masked the repeated-sampling variance a regression test depended on, something a plain threshold check has no way to catch. - CrewAI PR #6134: Fixed a bug where file tools leaked absolute filesystem paths in responses. A standard quality scorer never flags this because it only checks if the answer is correct, not what is hidden in the payload. These examples are part of a larger validation pass encompassing 29 PRs and 239 rows across LangGraph, CrewAI, and the OpenAI Agents SDK. The full data breakdown is available in the pr-analysis.md documentation on github. Instead of grading a single response, agent-eval statistically compares two whole batches of runs. By running versions A and B 50 times each, the tool executes: - A Mann-Whitney U test (requiring no assumption that scores are normally distributed). - A bootstrap confidence interval to confirm if the shift is real. - Cohen's d to determine if the shift is large enough to actually matter. A p-value under 0.05 by itself is not enough to call it a regression. The effect size is required for definitive proof. As part of statistical guardrails, two limits are utilized to ensure data integrity. - Under 30 runs per version: Triggers a warning due to a drop in statistical power. - Under 10 runs: Returns INSUFFICIENT_DATA rather than guessing at an answer. This is one piece of a larger observability, testing, and benchmarking infrastructure designed for production agents. We are already planning for the next version. Should AutoGen or the Vercel AI SDK be prioritized next? What do you suggest?