EvalCore - Snapshot-test AI behavior in CI
by•
EvalCore is a single-binary test runner for LLM apps and agents. Define cases and scorers in YAML, run local targets on every PR, and replay model or judge calls offline for $0. Use REST or shell targets, baselines, trials, model comparisons, and OTel traces.
Replies
Hi Product Hunt, I’m @abhishek_manyam. @kuladeep_mantri and I built EvalCore because an AI app can change after a prompt edit, model swap, or dependency update, and the failure often appears after release.
EvalCore makes the eval suite a normal repository artifact: one YAML config, JSONL cases, and one binary. Local shell targets run on each PR. Model and judge calls can be recorded in a SQLite cassette and replayed offline. A suite exits 0 or 1, so CI can block regressions without depending on live model calls.
EvalCore supports deterministic scorers, LLM-as-judge, custom scorers over JSON stdin/stdout, repeated trials, model comparisons, baselines, HTML reports, and OTel/OpenInference agent traces. Targets use HTTP or shell, so your app and custom scoring code can use any language.
The project is Apache-2.0. There is no account, hosted service, or telemetry.
We want feedback from engineers who already run evals in CI: where does your current setup become flaky, expensive, or hard to review? The quickstart takes about five minutes and uses no network. If you try it, tell us what target or scorer is missing.
GitHub: https://github.com/eval-core/eva...
Quickstart: https://evalcore.cc/getting-star...
Congrats on the launch, single-binary eval runners are a real gap. One thing that would help my team a lot: a built-in `evalcore diff` command that compares pass rates and cost across two runs and outputs a markdown summary. Right now I end up exporting CSVs and fighting pandas just to see what regressed on a PR.
@erdisleyma3t11 Thanks! And yeah, the CSV-plus-pandas dance is the exact thing we want to make unnecessary. You shouldn't need to stand up a data pipeline just to answer "did this PR make anything worse."
evalcore diff is a good fit. Runs already get stored with their pass/fail results and per-case cost, so pulling two of them and diffing pass rate and spend isn't a big lift. The part I'd actually fuss over is the markdown, so it pastes straight into a PR comment without looking like a wall of numbers. Probably per-case regressions up top, the total cost delta, and the noisier stuff folded into a collapsible section.
ran a small eval set against a local model and the offline replay feature actually works as advertised, no api costs during iteration. the yaml setup took a few tries to get right but the otel traces made it easy to see where things broke.
@melahatg2dw Love hearing this, the offline replay is the feature I most wanted to nail, so it's good to know it held up on a real local-model run. Zero API spend while you iterate is the whole point.
The YAML taking a few tries is the part I want to poke at, though. What tripped you up? If it was error messages that pointed at the wrong line, or a field that wasn't obvious from the docs, that's on us to fix. I'd rather the first config just work than have people brute-force it.
And good to hear the OTel traces earned their keep. Seeing where a run broke without adding print statements everywhere was exactly the hope. Thanks for actually kicking the tires on this.
Ran a few of my agent evals through it locally and honestly the offline replay thing is a lifesaver, no more burning cash re-running the same judge calls during debugging.
@tlaye4ri Ha, "burning cash re-running the same judge calls" is basically the sentence that got this feature built. Judge calls are the worst offender since you're paying for a model call just to score, and during debugging you re-run them dozens of times without the inputs actually changing. Caching those was the obvious win.
Glad it's holding up on real agent evals and not just our toy examples. If you hit a case where the cache misses when you expected a hit, tell me, that's the kind of bug I want to squash early.
The YAML setup is clean and getting a full eval suite running on a PR took maybe ten minutes. Really appreciated being able to replay judge calls offline without burning tokens.
@melisterze Ten minutes to a full suite running on a PR is exactly the bar we were aiming for, so that's great to hear. The YAML being clean took a few rewrites of the schema to get there, so I'm glad it landed.
And yeah, replaying judge calls offline is the one people seem to feel immediately, since that's where the token bill actually hurts. Thanks for giving it a real run on your own setup.
If anything in the config felt clunkier than it needed to be, I'm all ears. The ten-minute setup is the number I want to keep dragging down.
Love the offline replay feature, that alone saves me a ton of money on judge calls. One thing that would make it even better for my workflow is a built-in web UI for diffing trial results side by side, so I can quickly spot regressions when comparing models without grepping through JSON output.
@rorbay74131 Glad the replay's earning its keep. The regression-spotting need is real, though I'd probably solve it with a diff command plus a self-contained HTML report you can open locally or attach to a PR, rather than a full web UI, since the whole point of EvalCore is one binary with no server to run. Same "spot it in five seconds" outcome. Would static output cover your case, or is there something about a live UI you'd actually miss?
having replays for offline runs is genuinely useful, would love to see a side-by-side diff viewer for two eval runs so you can quickly spot which cases regressed or improved without digging through json.
@bulemglezgjgz Glad the replays are useful. Diffing two runs to see what regressed or improved is a real gap right now, and I'd rather you not dig through JSON for it. I'm leaning toward a diff command that prints a side-by-side, plus a self-contained HTML report you can open locally or drop on a PR, since keeping it to one binary with no server is kind of the point. Would that cover it, or is there something about a live viewer you'd actually want?