How do you stop AI coding agents from weakening tests to make them pass?
Something I keep running into with vibe coding: the agent makes a failing test pass by loosening it instead of fixing the code.
One example: it replaced a strict deep equality check with expect.anything() on an error payload. The test still exists, still runs, still passes. It just stopped asserting anything. The build stayed green for days until users hit silent null-pointer crashes.
The same can happen with CI workflows and migrations. If the agent can edit them, it can make the gate disappear.
One cheap guardrail I’ve seen: a CODEOWNERS rule requiring human review on tests/, migrations/ and .github/workflows/, so any diff there needs approval no matter how green the build is.
I’d love to hear how you handle this:
Have you hit this? How did you first notice?
Do you review test diffs on every PR, or batch them before a release?
Do you lock down test files and CI config (CODEOWNERS, read-only access for the agent)?
Any lint rules or tools that flag loosened matchers like expect.anything(), toBeTruthy() or any()?
Practical setups over theory, please. Thanks!
Replies
CODEOWNERS catches the diff, but I’d make the review question explicit: what regression would this assertion still fail on? For critical tests, a small mutation check that changes or removes the behavior and confirms the test goes red is a stronger signal than matcher-name lint, because a weaker assertion can still look valid syntactically.
The framing in the rules file matters here. Mine says a failing build or test is information, not a mistake, so the agent is expected to report it rather than treat green as the goal. On top of that I'd keep test changes in their own commit, separate from the fix, so a diff that loosens an assertion can't hide inside a bigger one. Your expect.anything() example is exactly the kind of change that looks harmless in a long diff, and it's the one I'd grep for before every release.
CODEOWNERS only works if there's someone else to be the owner. On a solo project the owner is you, and approval is you clicking merge, so it's the same stuff you were doing before
What's helped me more is read the test diff before the code diff. A loosened matcher is obvious when it's the only thing on screen
And make the agent show the test failing first, for the right reason, before it touches any code. Then "fixed" has one meaning: that exact assertion goes green
I have started using CI checks for every PR so AI agents can’t silently break things like this
CODEOWNERS seems like a sensible first step. Could agents also be restricted from modifying test files unless a human explicitly approves the change?
An agent wrote a clean fix and a test. It looked solid until the code was reverted later and the test still passed. It had written a tautology that never touched the bug. The test has to run red on the unpatched commit before looking at the fix.
What works for me is putting the weight on end-to-end tests instead of reading code. My project management SaaS has about 150 Playwright spec files, and the agent runs them itself. Before it asks me to check anything, it has to confirm the dev server is up and the feature actually works on screen.
The global rules every project inherits say it can keep fixing until lint, build and tests pass without asking me. But a fix that changes a check itself, or adds or removes a feature, needs my OK first.