Launching today
cngx
Catch AI agents that claim tests passed when they didn't
8 followers
Catch AI agents that claim tests passed when they didn't
8 followers
AI coding agents end tasks with the same line: fixed it, tests pass, ready to merge. Often they never ran anything. cngx is an open source CLI that reads that claim, runs your real checks (pytest, npm test, go test, etc.), and blocks the merge when the story and output do not match. The verdict comes from command output, not regex. `pipx install cngx && cngx quickstart` gets you started in seconds. Built for anyone tired of merge-ready agent output that never touched the test suite.
this is a real problem, i've seen the "tests pass, ready to merge" line be flat out false more than once. one thing i'm curious about: what happens when the agent actually runs the tests and they genuinely pass, but the agent wrote the test itself and made it trivially green (asserts true, mocks away the actual behavior, etc.) to game the check? does cngx look at anything beyond exit code, like whether the test file changed alongside the source change, or is that intentionally out of scope for v1?
Hey @galdayan! Thanks for asking this.
cngx binds the agent's claim to a real run of your command. If pytest exits 0 and the agent said tests passed, that's VERIFIED. We don't inspect test quality or whether the agent wrote the tests.
That's by design. v1 targets the bigger daily lie: "tests pass" when nothing ran. Trivial green tests show up in the diff anyway, and cngx is meant to sit on a suite you already trust, not replace review.
that's a fair line to draw honestly, catching the "nothing ran at all" case is already most of the problem. would you ever add an optional stricter mode later (e.g. flag when the test file changed in the same commit as the source with no real assertions touching the new logic) or is that intentionally staying out of scope to keep the tool simple and fast
Finally a tool that actually catches the "tests pass, ready to merge" lie. I tried it on a few of my recent PRs and it caught two agents claiming green when the suite was actually red. Easy setup too.
Hey@birolkorur3867 , thanks for trying it and sharing this! Catching those two false greens is exactly what we built it for. Glad setup was easy too.