The moment you realized your AI agent couldn't be trusted
by•
Mine once created a reminder for a meeting that didn't exist, then flagged it three days in a row. I only caught it because I kept trying to remember who I was supposed to be meeting.
The interesting part is I still use it for the same task; I've just learned which of its confident moves to double-check. It feels a bit like working with a sharp new hire who's still finding their footing.
When did you first notice your AI agent needed a second look, and did you stop using it or just build a workaround like I did?
96 views
Replies
@doganakbulut what's the stranger thing your AI agent has ever done?
@doganakbulut Great Question
Initially, an agent would name a file as per need and move between folders, I couldn't figure out the reason why. STRIKE #1
The day Claude started tokenmaxing by committing to Git, in spite of being told not to. STRIKE #2
Once I wanted to do deep research using Gemini, upload the PRD, and let it do its thing. The output felt fishy; none of the content I expected was there. So I asked, "What is the problem?" The reply was frivolous. "I was not able to open the prd, so i went ahead with generic research"—STRIKE #3
The shift for me was realising confidence and accuracy are completely uncorrelated in these systems. I work in data and you learn quickly to distrust a clean looking number faster than a messy one. Agents are the same. The failures that hurt are never the obvious errors, they are the plausible ones that sail straight past review.
My workaround is to grade the evidence, not the answer. Make it show the query it ran, the file it read, the source it pulled from. If it can name the source you can verify in seconds. If it cannot, that is usually the tell. I never stopped using them, I just stopped marking the output and started marking the working.
@oshylabs @doganakbulut Arnold’s point about confidence and accuracy being uncorrelated is the big one for me. Humans usually show uncertainty, but these tools don’t. They can sound equally confident whether they’re giving the right answer or inventing a meeting that never happened.
That’s what makes the believable mistakes the hardest ones to spot.
@doganakbulut @jared_salois The part that took me longest to accept is that the tone carries no information at all. A human hedging tells you something. These systems hedge when their training data hedged, not when they are unsure. Once you stop reading the tone and start asking for the working, the confident wrong answers lose most of their power.
@doganakbulut Two habits from data work carried over. First, anything that feeds a decision gets traced to its source before it gets used. One question does most of the work: where did this come from? If the agent can point at the query it ran, the file it read or the URL it pulled, I move on. If it paraphrases instead of pointing, that is the flag. Second, I keep a short list of the failures that have actually cost me time, invented meetings, invented requirements, quietly skipped steps, and I check new output against that list first. It is boring but it catches more than any clever prompt has.
@oshylabs "Grade the evidence, not the answer" is exactly the thing I ran into building FounderFlow (I'm the founder, disclosing since it's directly relevant here). Early versions just gave me a flag saying "this needs attention" with total confidence, and I trusted it right up until it was wrong about something that mattered. What actually fixed it wasn't a smarter model, it was forcing it to rate its own confidence honestly, Verified vs Very Likely vs Needs Review vs Monitor Only, instead of one flat answer. I still override about 1 in 6 flags, but now I know which ones to double check instead of trusting all of them equally.
I don’t think trust with agents is binary, for me it’s more about knowing which tasks need approval gates. if the agent is drafting, researching or organizing, i’m fine with speed. If it is creating events, sending messages, changing data or touching anything external, i want a checkpoint before it acts.