Launched this week
Agent Eval Kit
Deterministic offline release evidence for AI agents
8 followers
Deterministic offline release evidence for AI agents
8 followers
Turn an AI-agent evaluation into one portable, machine-verifiable acceptance record. Agent Eval Kit seals the exact JSON contract and evidence, pins the commit and command, adds a canonical SHA-256 digest, runs eight fail-closed checks, and exits deterministically in CI. It works offline with zero telemetry and no production credentials. Version 1.0.0 is sold as a one-time USD 1,000 Team License for one legal entity and up to 25 internal users.
Everyone wants AI agents in production, but very few teams have confidence in how they evaluate them.
Curious—what's the most common false sense of confidence you've seen that your evaluation process exposes?
A small thing that would help a lot - add a GitHub Action example in the README showing exactly how to wire the eight fail-closed checks into a PR workflow, since most of my team lives in GitHub and a copy-paste yaml snippet would cut the setup time considerably.
Finally a tool that treats agent evals like a real artifact rather than a vibes check. Loved that the SHA-256 digest and sealed JSON make it trivial to prove what was tested and against which commit.
love that the SHA-256 digest and pinned commit turn the whole eval run into something auditors can actually verify later instead of trusting screenshots