Have you ever asked a second AI because you didn't like the first one's answer?
I've caught myself doing this more often than I'd like. Especially when I'm using AI to think through an important decision.
Ask ChatGPT something important. Get an answer. Not completely convinced, so ask Claude. Maybe try Gemini too.
And most of the times, I don't necessarily trust the best answer. I start gravitating towards the answer that makes the most sense to me. Which is potentially just confirmation bias with more compute.
It made me rethink how I use AI for decisions. Maybe asking more models isn't enough. The real value would come from making them disagree, challenge assumptions and expose what the others are missing before I decide.
Has anyone developed a system for doing this well? Or do you still mostly use one AI and trust your own judgement at the end?
Replies
ClawTeams
i do a lot
but the problem is nowadays all the major models look the same, i mean there is not much difference among answers by major llms
we need a more diversified Model World
@tony_tan10086 That's exactly what I've been noticing too. Even when the models differ in reasoning, the final recommendation often converges to something very similar.
Which makes me wonder if we need more diversified models, or a better way of forcing existing models to take genuinely different positions, challenge each other, and defend their reasoning.
Otherwise asking 5 models might just become 5 versions of roughly the same answer.
Softorino 💻📲
I do SEO, so I get to check the models against something real, and the pattern I see is that the second one is usually confidently wrong in the same direction as the first. What killed the confirmation bias for me was not another model but a number from outside, like actual search volume or what Search Console already logged. How do you handle the decisions where there is nothing external to check against?
@yelyzaveta_kibets That's the harder problem. And it's the one I've actually been sitting with. When there's an external truth, like search volume, revenue or Search Console data, you can eventually test the answer against reality.
But what about decisions where there isn't one? Should I hire this person? Enter this market? Change my pricing? Walk away from a client?
I've been experimenting with something different. Instead of asking multiple models the same question and comparing answers, I give them different perspectives and make them challenge each other's reasoning, assumptions and blind spots.
It still won't give you "the right answer." Maybe that's the wrong thing to expect from AI anyway. Before a consequential decision, I don't want certainty. I want a better sense of how I could be wrong.
Softorino 💻📲
@vineeth_n1 That reframe helps. The SEO calls I can check against Search Console eventually, but the ones I can't check are the ones I actually lose sleep over, like what to charge. When the models are arguing rather than agreeing, what tells you the argument is finished?
I usually ask a second model to challenge the first. Does anyone have a good workflow for making that comparison more objective?
@albertnelson I've been experimenting with this, and I'm starting to think comparing the two answers isn't actually the most useful part. What seems to work better is separating the roles: one model makes the case, another actively looks for flaws and hidden assumptions, and then a third evaluates both arguments against the original context. The tricky part is making that last evaluation objective. Without external data, you're still asking an LLM to judge other LLMs.
So I've been focusing less on "which answer won?" and more on: what did they agree on, where did they disagree, which assumptions are driving that disagreement, and what could make the decision fail? Still experimenting with the workflow though.