Does your AI reviewer ever tell you it does not know?
I asked an AI assistant to review a repo before shipping it.
It came back with a clean report, a confident tone, and a long list of passes. Then I checked one of the lines it cited.
The line said something else. The mistake itself didn't bother me that much. What bothered me was that there was no shape in the output for "I don't know."
Everything was either a problem or a pass. And the passes were doing a lot of unearned work.
So I changed the rules: if the code couldn't actually tell us something, it had to say UNKNOWN.
About half the checklist came back UNKNOWN.
Rate limiting at the edge. Whether backups have ever been restored. Things the code genuinely cannot tell you.
The report became much less satisfying—and much more useful.
Do you get honest uncertainty out of your AI coding tools? If so, how did you get them to admit when they don't know?
Replies
The backup example is a good one. having the backup code there doesn't really prove you can actually restore everything when you need.
Kinda funny that making the report look worse actually made it more trustworthy. i'd take an honest unknown over a perfect looking report any day.
One bad citation would make me question the whole review, tbh. If that slipped through, what else did it confidently get wrong?
The backup rstore example is a good one. Would you want the reviewer to automatically flag things like this as requiring a manual verification step?
I think having an unknown option is actually a good thing. if it doesn't have enough info to know, why force it to say pass or fail?