We gave 5 frontier LLMs the same 1,000 fact-checks. They disagreed on 63%.
We’ve been looking at a question that feels increasingly relevant as more products put LLM output directly in front of users: if two frontier models are asked to judge the same factual claim, how often do they actually reach the same conclusion?
We tested five frontier models on 1,000 real-world claims under the same forced-choice setup.
On 63% of the complete claims, at least one model disagreed with the others or no majority formed.
On 23%, the disagreement was substantive — verdicts at least two categories apart.
This doesn’t tell us which model is right on an individual claim. It tells us something different: models with similar benchmark standing are not interchangeable as factual judges.
What does that mean for AI products? For people building AI products, we are curious how you handle this today.
If your system needs to act on factual output, do you trust one model, ask another model to review it, use consensus, retrieve external evidence, escalate to a human — or something else? Let us know in the comments.
Full study + dataset: lenz.io/research/llm-disagreement
Replies
consensus was the first thing i tried and its the thing that burned me. i had two claims to verify today and asked a second model to check them. it agreed with both, and both were wrong in the same direction, becuase it was reasoning from the text i handed it rather than from the source. what caught it was leaving the models entirely and fetching the url logged out. one of the two things i was certain about returned a 404 to anyone who wasnt me. so the rule i run now is that a model can flag a claim but it cant clear one. clearing needs something outside the model, a fetch, a db row, a log line. i also tell the reviewer to default to refuted when its unsure, otherwise it agrees to be agreeable. on your setup, is the 63% mostly cross-family, or do models from the same lab disagree with each other at a similar rate? that would change what consensus is worth at all.
Lenz
@rabnoor_s Hello, I am one of the co-authors of the paper. This is a great example why we should not treated the majority on some claims as a ground truth. In our future work we are planning to make a human labeling on every claim and then to measure the accuracy but for now we are measuring only the disagreement between the models. On your question: the 63% disagreement is entirely between cross-family models. We picked the latest models of 5 of the most used platforms and we ran 1000 claims submitted from users on every model. It will be also interesting to measure the disagreement between models from same provider especially because that will help separate differences caused by model families from those caused by retrieval and verdict calibration.