We spent the summer testing an AI reviewer. What should block a PR?

We have been quiet here through most of the summer because we were heads-down testing our reviewer on real pull requests. After using it across 60 PRs, one lesson stood out: a finding is only useful if the developer can see why it should be trusted.

In QualityMax, every review card records the analyzed commit, model, re-review number, and one of four proof levels:

VERIFIED: an independently judged patch

REPRODUCED

GROUNDED: deterministic evidence

MODEL-ONLY

We also re-judge every finding before showing it, retire stale findings after a fix, and never auto-apply a proposed patch. For confirmed critical findings, the reviewer can draft up to three patches, ground them in the PR head, and have an independent judge score them. The merge decision stays with the developer.

The hard question is where to draw the line. Should a MODEL-ONLY finding be allowed to block a PR when the issue is security-sensitive, or should blocking always require stronger evidence?

Full methodology: https://qualitymax.io/blog/blocked-by-our-own-reviewer