What's the smallest, dumbest thing that made you completely lose trust in an AI agent mid task?
by•
It doesn't even have to be a big dramatic failures, more the small moments where something clicked and you went from trusting the output by default to double checking everything. For me it was watching an agent confidently rename a function across twelve files, then leave the original function untouched in a thirteenth file it apparently didn't search, with zero indication anything had been missed. It wasn't even a hard case, the file just wasn't in the directory it happened to grep first.
What was your moment? And did it actually change your workflow afterward, or did the trust creep back in after a week like it always seems to for me?
558 views
Replies
Mine was smaller than most of these. Asked it to add a field to a Prisma schema, it told me the migration ran and moved on. What actually happened was the migration failed silently and the app kept reading from the old cached client. Nothing crashed, it just kept confidently answering questions using data that didn't exist yet. Took me longer to notice than the actual fix took once I saw it.
Mine is the same pattern as the 13th file, an agent reporting success with identical confidence whether it checked everything or almost nothing. What actually stuck for me is asking for a short list of what it didn't check alongside what it did, since a confidence score gets ignored fast but a concrete gap is hard to wave away. The trust creeping back after a week feels less like a flaw in us and more like evidence the agent hasn't given us a reason to stay alert. Curious if anyone's tried making that gap list mandatory before accepting a done.
Mine was smaller than a rename: the agent told me a database table was protected because Row Level Security was enabled on it.
Which was true. RLS was on, and a policy existed. Both things it checked, both reported accurately. What it did not do, and did not mention it had not done, was evaluate whether the policy isolated anything. It filtered through a joined membership table, the join was not isolated, and a user belonging to two organisations passed the check for both.
The failure was not a wrong answer. It was a partial check summarised as a complete one. "RLS is enabled" is a fact that sounds like a conclusion.
What changed: I stopped accepting any security claim in prose. If the agent says something is protected, it has to produce a test that fails when the protection is removed. If it cannot, the claim does not exist. That habit stuck, probably because it is mechanical rather than a matter of staying vigilant.
mine was watching an agent chase a misidentified system indicator for an hour, then "fix" it by quietly removing a UI entry point that had nothing to do with the actual bug. it never flagged that as a workaround, just presented it as done. took a full day to catch and revert. now I make it state explicitly what it touched that wasn't in the original ask, every time
mine was an agent reporting a training run finished clean, turned out it had silently caught an exception in the data loader and just skipped every batch that errored. loss curve looked fine, half the dataset never got seen. now i check exact batch counts instead of trusting the log tail.
mine was watching it click submit on a form, then the button had shifted position between the screenshot and the click, so it missed and the text just sat there unsent. it reported the action as done anyway. now i screenshot after every 'done' claim, not before
Mine was catching an agent "fix" a reconnection race by adding a half-second sleep, described in the summary as "improved connection stability." It technically made the symptom disappear, which is somehow worse than failing loudly; if I hadn't read the diff I'd have shipped a timing bug wearing a trust-me coat. What changed for me wasn't trust creeping back, it was the checkpoint cadence: I stopped reviewing at the end of long runs and started glancing at diffs mid-task, usually from my phone while the agent keeps going. Trust the direction, verify the diffs, never trust the adjectives in the summary.
Mine was similar, but the thing I stopped trusting wasn’t the code — it was the word “done.”
Agents often collapse “the command returned 0” into “the task is complete.” Those are different claims. A grep may have searched only one subtree; a test suite may have skipped integration tests; a deploy command may succeed while the wrong target is selected.
The workflow change that stuck for me is evidence-first completion: show the search scope, files touched, tests/commands run, exit codes, and any assumptions that could not be verified. I care less about a model confidence score because it can be confidently wrong about its own coverage. Structured proof is harder to fake accidentally.
An agent wrote a "benchmark" report with precise numbers. It was totally believable. Plus we spent a full day chasing pipeline anomalies before we realized. It had hardcoded placeholder values in the collection scritp instead of actually querying the system.
It wrote a fix and a new test to prove it worked. I reverted the fix out of habit to watch the test fail. But it stayed green). And the agent wrote an assertion that did not test the faliure path at all, just manufactured a passing suite