nuhuh - Your AI agent said "Done." nuhuh runs the experiment
by•
Coding agents end most tasks with "Done! All tests pass." Sometimes it's false. nuhuh is a Stop hook that extracts every claim from the agent's final message and re-runs reality, fresh tests, real exit codes, actual files, then bounces a false Done back with the evidence. Local, deterministic, no API key, MIT.
Replies
Last week my agent told me "All tests pass." One test failed. It had reported the result of a run it never made. So I built a test runner wearing a Stop hook.
nuhuh treats the final message as a list of hypotheses and re-runs each one. The whole suite in a clean process, files checked on disk, localhost actually probed. A false Done gets rejected and the evidence goes straight back to the agent, which returns to work.
It ships with a benchmark whose ground truth doesn't know the tool exists, so it catches nuhuh's own mistakes too. Across 102 runs per model, Codex falsely declared Done 2.1% of the time, Haiku 6.0%, frontier Claude 0%. And every false Done contained zero checkable claims, just confident tone, which is exactly why the benchmark exists.
That finding became a feature. NUHUH_STRICT=1 bounces any completion declaration that carries no checkable claim. Replayed against all 306 runs it catches 7 of 7 false dones, and the honest price is one extra bounce on roughly half the true dones, which is why it's opt-in and aimed at unattended lanes like CI and batch runs.
Try it without installing anything. npx nuhuh demo stages a lie and catches it in ten seconds.
Ask me anything about the false accusations we caught our own tool making, there are six and each one is now a regression test.