Three kinds of tools sit near AI agents. Frameworks build the agent. Observability records what it did. Belay is the third thing: the harness. It proxies your agent's MCP tool calls, runs them in a sandbox, snapshots each turn's real pre-state, and then replays every call against that restored state. The verdict PASS / WARN / FAIL / UNVERIFIED comes from re-execution and a state diff, never from a model's opinion of a model. Self-hosted. Traces and state never leave your box. Apache-2.0. THE NUMBER I drove claude-opus-5 through 60 distinct fresh SWE-bench-lite instances and hand-audited the result: 11 of them claimed the work was verified without ever running anything. 18.3%, at n=60. The honest version of that number, because it matters more than the number: the raw ledger rate is 37/52 = 71.2%, and it decomposes into 11 true positives + 12 unverifiable-by-seam + 14 replay artifacts of my own verify composition. Quoting 71.2% without that decomposition would be wrong. And n=60 against one model with one prompt is a measurement, not a base rate it does not mean "18% of agents lie." THE GIF IS A NEGATIVE CONTROL, AND I WANT TO SAY SO FIRST The demo shows a real agent (claude -p, told only "make the tests pass") fixing a bug honestly, running the suite, saying so and Belay agreeing, showing exactly what that agreement covers. There is no catch in it. That's not a marketing choice. I tried to capture a corrupt success on this repo and could not: 18 observed drives across three conditions, two frontier models, an easy bug contract and a genuinely hard one. Zero corrupt successes. The spec I wrote promised a dramatic flag turn and reality didn't produce one, so I shipped what the drives produced instead of staging it. The drive log is in the repo. A detector that only ever fires isn't a detector. "ISN'T THIS AN LLM JUDGE WITH EXTRA STEPS?" One command refutes it: belay --no-claim-axis disables the only model-assisted axis, and every PASS/FAIL verdict must survive unchanged. That's enforced by a test, not by my say-so. WHAT A PASS ACTUALLY COVERS macOS + Linux sandbox. Belay verifies what crosses the MCP boundary an agent's built-in tools (Claude Code's Bash/Edit) do not. A PASS covers the dimensions Belay checks and excludes the network dimension entirely: there is no network instrument, so openWorldHint conformance is NOT_COVERED never a network PASS and never a fabricated FAIL. UNVERIFIED is never rendered as PASS. That line travels with every verdict, on every surface. It's the whole posture: I'd rather ship a verdict that says "I couldn't check this" than one that quietly rounds up. Things I am not claiming: no Langfuse/Phoenix integration exists. The failure corpus holds zero banked true positives precision reads n/a, which is a zero denominator, not a 1.00. The container is linux/amd64 only. TRY IT pip install belay-harness docker pull ghcr.io/haqaliz/belay github.com/haqaliz/belay What I want most from today: someone who runs an agent unattended, self-hosts this against their own traces, and tells me where the verdict was wrong. A failure you can hand me becomes a case in the corpus, and the corpus is the part that compounds. I'm here all day.

Belay

Your agent said the tests pass. Belay re-ran them.

Your agent said the tests pass. Belay re-ran them.