agent-eval: Statistical regression testing for LLM agents

Hi Everyone!

@rudrendu_paul_ai and I are officially soft-launching a new OSS solution, agent-eval, today.

When we build AI agents, we constantly run into a persistent pain point: we change a prompt or swap a model, and while our standard evals still pass, the agent's actual behavior shifts anyway.

This happens because standard threshold checks only evaluate isolated responses, asking,

"Did this single output pass the test?"

They never ask,

"Did the overall distribution of behavior fundamentally shift between versions?"

We built agent-eval to replace anecdotal guesswork with rigorous statistical regression testing. It runs your agent 50 times on Version A and 50 times on Version B, replacing binary pass/fail metrics with rigorous statistics.

We designed the output to provide:

• A Mann-Whitney U p-value.

• A Cohen's d effect size.

• A bootstrap 95% Confidence Interval to prove whether the behavioral shift is real or just noise.

This OSS solution is entirely open-source (Apache 2.0), self-hostable, requires zero API keys, and is built on pure Python + SciPy.

Test it out before the official launch:

pip install agent-regress-cli

Repo:

https://github.com/RudrenduPaul/agent-eval

Please drop any questions about the stats methodology, the codebase, or agent testing in the comments below! We're happy to answer them.