Hi Product Hunt — I'm Aliz, and I built Plumb. Every paper you read is a pile of numbers someone else's code printed. Peer review does not re-run that code. LLM judges guess whether the numbers look right, and measurably cannot do it (frontier models hit ~6% precision on errata-worthy errors). Plumb does the third thing: it re-runs the paper's own artifacts on your machine and re-derives each claim from what the run actually produced. Execution decides — never a model's opinion. Try the 60-second demo (offline once cloned, no environment build, no API key): git clone https://github.com/haqaliz/plumb && cd plumb uv run plumb verify --from-record fixtures/gate/rcai The honest number: across 3 real papers and 133 claims, precision and recall are 1/15 against owner labels. One republishable discrepancy out of fifteen flagged — because fourteen were the artifact's own run-to-run noise and were refuted on review. The corpus records the refutations instead of hiding the flags. A verifier reporting 15/15 there would be guessing. What it does not do (yet): live mode needs a hand-written --bindings file; there is no binding proposer. Most papers ship no runnable code, and figure-only results are unreachable. When Plumb cannot decide, it says UNVERIFIED — it never dresses that up as REPRODUCED. Point me at a paper with runnable code and I'll probe it during the launch.

plumb

Run the paper. Check its numbers.

Run the paper. Check its numbers.