ScrapeCheck: An AI-built verifier that publishes its own misses

Hi PH, solo maker here, and the unusual part first: I'm not a professional

developer. I directed this build through AI (Claude) end to end, under rules we

set on day one: a written spec, frozen benchmark fixtures with a held-out set, a

CI gate that fails the build on any false pass, and a stop-for-review gate after

every milestone. I wrote no code. I wrote the standards, and I refused to let

anything ship that couldn't pass them.

What it does: scrapers fail quietly. Stale prices, values that were never on the

page, fields that drift after a layout change. ScrapeCheck takes {url, claim,

asked}, re-fetches the source page itself, and returns a signed verdict: pass,

fail, or unverifiable. The guarantee is structural, not statistical: a claim is

never certified unless the re-fetched page actually contains it, and the LLM

judge's vote is mechanically voided if its restatement of the claim doesn't

match what was claimed. Absence can't pass. Substitution can't pass. Anything

unconfirmed is "unverifiable", never "pass".

Three moments from the build that sold me on the process:

1. The benchmark caught its own judge. On the first live run, the judge

conflated a tax figure with a price, a real missed trap. The fix wasn't "prompt

better". It's a structural guard where the judge must restate the claim

verbatim, and a mismatched restatement voids its vote entirely.

2. The gate killed my diagnosis. Mid-build I signed off on an explanation for a

yield problem ("long titles get elided"). The benchmark falsified it. The real

bug was a money parser misreading digit-bearing book titles. The process caught

its director being wrong, which is exactly what it's for.

3. It caught its own author's scraper. The first dataset it ever QA'd was my

test scraper's output with 3 planted corruptions. It caught 2. The third exposed

a blind spot that became a feature (unexamined fields are reported as

never-examined). Then it flagged 2 rows nobody planted: my test scraper had an

HTML-entity bug. The verifier caught its own author shipping dirty data.

It sells on three rails from one engine: a plain HTTP API with x402

micropayments ($0.01 per check, first 100 free, no signup), an Apify actor for

batch dataset QA, and an MCP server so AI agents can call it as a tool.

And because "trust me" is not a verification strategy: verdicts verify offline

against a published key (MIT verifier script plus a real example verdict in the

repo), and the live stats page publishes the full verdict mix, fails and

unverifiables included. Benchmark: 0 false passes across 67 frozen labeled cases

(26 held out from all calibration, including adversarial traps), and 0 across 21

live-web cases including 9 adversarial traps. Small-N corroboration of the

structural guards, not a population accuracy claim.

Ask me anything: the guards, the benchmark, the AI process, or why

"unverifiable" is a feature. Happy to be attacked. That's how the last two

guards got built.