Open Source Evals Launch
Ahead of Product Hunt we have open sourced Prefactor Evals, No LLM Required. Apache 2.0, public repo.
Most agent failure is behavioural: a silent loop, a step that failed, a run that never finished, a task that took four times the usual work. All of that is checkable in code, deterministically, for nothing. There is no model client anywhere in the project and there never will be, so it cannot cost you a token by construction.
7 checks run on defaults with no config: errors, loop detection, redundant tool calls, termination state, latency budget, efficiency, conversation length. 5 more are optional. Efficiency and conversation length calibrate against the median of that agent's own recent runs, so they catch an agent getting worse rather than differing from someone else's ideal.
Live mode is the part I care about. It grades a run span by span while it is still going and can stop one that breaches. A loop trips on the offending call, not after the damage. Terminate is never the default.
Python and TypeScript, held to identical results by a conformance suite that diffs every status, message and value across both. Any divergence fails CI.
Replies