How do you trust tests written by the same model that wrote the code?
Basically all our code gets written by agents now, and so do the tests. Usually same session, same model. And I keep hitting the same loop:
Agent builds the feature
Agent writes the tests, everything green
Ship it
A user hits an edge, I dig in, and the tests were checking what the code does, not what the ticket asked for. Suite was green the whole time.
The annoying part is that if the model misread the ticket, its tests misread it the same way. It's basically marking its own homework. The model is just agreeing with itself.
So what do people actually do here?
Different model writing tests than the one writing code? Does that actually help or do they just share the same blind spots?
Tests written from the ticket before the agent touches any implementation?
Humans only do the test files, agent does everything else?
I'm building in this space, so half the reason I'm asking is to find out whether everyone's setup is as screwed as ours was or if someone's actually cracked this.
Keen to hear what's working - thanks
Replies
Me thinking about your edge case example makes me believe acceptance criteria their own tests before implementation begins. Would a mandatory ticket review step catch those hidden gaps earlier?
@logan_hogg7 Would separating planning from implementation reduce that shared misunderstanding begore anything gets generated?