Does your AI reviewer ever tell you it does not know?
I asked an AI assistant to review a repo before shipping it.
It came back with a clean report, a confident tone, and a long list of passes. Then I checked one of the lines it cited.
The line said something else. The mistake itself didn't bother me that much. What bothered me was that there was no shape in the output for "I don't know."
Everything was either a problem or a pass. And the passes were doing a lot of unearned work.
So I changed the rules: if the code couldn't actually tell us something, it had to say UNKNOWN.
About half the checklist came back UNKNOWN.
Rate limiting at the edge. Whether backups have ever been restored. Things the code genuinely cannot tell you.
The report became much less satisfying—and much more useful.
Do you get honest uncertainty out of your AI coding tools? If so, how did you get them to admit when they don't know?
Replies
no, and we stopped asking. ours drafts comments and has never once said this one is weak, they all come back fine.
what worked was not asking it to rate its own output. we record afterwards which ones actually got flagged and feed that back in. the check had to sit outside the model.