What's the smallest, dumbest thing that made you completely lose trust in an AI agent mid task?
by•
It doesn't even have to be a big dramatic failures, more the small moments where something clicked and you went from trusting the output by default to double checking everything. For me it was watching an agent confidently rename a function across twelve files, then leave the original function untouched in a thirteenth file it apparently didn't search, with zero indication anything had been missed. It wasn't even a hard case, the file just wasn't in the directory it happened to grep first.
What was your moment? And did it actually change your workflow afterward, or did the trust creep back in after a week like it always seems to for me?
139 views
Replies
For me, it was watching an agent confidently generate code using APIs that didn't even exist . The confidence was more concerning than the mistake . Now I verify assumptions instead of trusting fluent outputs.
@terranicgerald The moment it silently deleted a small piece of logic while refactoring . Everything still compiled, but a key edge case disappeared. That's when I realized "no errors" doesn't always mean "no problems."
Mine was when an agent said it had “checked all relevant files” and then missed the exact file I had explicitly mentioned in the prompt. Nothing dramatic, but after that I started treating confidence as completely meaningless. Did your workflow change toward requiring explicit search summaries?
@ella_reyes1 That experience got me generating a JSON dependency graph that tracks which edits cascade across files, so nothing slips through. I regenerate it at the root after any major change in the codebase with a script I wrote for it.
Mine was a tiny one it ignored a constraint I had repeated twice. After that, I started checking every “done” myself.
@joseph_parker3 good thing that you do a manual verification
File and folder names were being referenced in a way that caused the AI to interpret them as code logic rather than application identifiers. For example, we have an application named test_packages, which required updates to the relevant skills to ensure it was handled correctly. A similar issue occurred with rate_limit, which can refer both to a security control checkpoint and to an application within the platform.
@francois_marais_nz That's a subtle one, and kind of a nasty class of bug because nothing actually throws an error, it just quietly reasons about the wrong thing. Naming collisions like that are almost worse than a clean failure since the output can look plausible while being built on completely wrong assumptions
The moment for me was an empty diff followed by a very confident summary of changes. Since then i trust the diff more than the explanation.
I once watched an agent fix the obvious error while quietly creating a new dependency problem. How much autonomy should agents really have when the fix has side effects?
@drew_dunham My rough answer is that autonomy should scale with blast radius, not with how confident the agent sounds. Fixing a typo in a log message, sure, do whatever. Anything that touches what a piece of code depends on or is depended on by, that should require a second look before it's considered done, even if the original bug is fully resolved.
That's a great example because the mistake wasn't the rename it was the confidence. Do you think AI agents should start surfacing uncertainty by default instead of presenting every result as equally complete?
@amberburch399 Yeah, that's exactly it, the rename itself was trivial to fix. What actually cost me was that it reported success with the same confidence whether it had checked one file or all thirteen. There was no signal to tell those two situations apart.
I like the idea of surfacing uncertainty by default, but I wonder if it'd just get tuned out fast. If every response has some kind of confidence caveat attached, you stop reading them the same way you stop reading cookie banners. Feels like it'd need to be more targeted, maybe something like flagging specifically when a search or edit came back with fewer matches than expected, rather than a generic disclaimer on everything.
I lost trust when it confidently said a command succeeded… while the terminal clearly showed an error. That one was almost funny.
@rose_florean Curious what agent you were running when that happened, always interesting to see if this shows up more with certain setups.
Mine was an agent fixing the bug but removing the logs that explained why the bug happened. Technically successful, practically frustrating.