ChatGPT vs Claude: which hallucinates less?

The honest short answer

There is no stable winner. On Vectara's HHEM leaderboard (announced November 19, 2025), which measures factual consistency on a summarization task, both GPT-5 and Claude Sonnet 4.5 exceed a 10% hallucination rate on Vectara's newer, harder dataset of over 7,700 long articles (Vectara, Nov 2025). Vectara notes this measures one summarization task, not all AI use, and small differences between closely ranked models should not be over-read.

So treat any headline "X hallucinates less than Y" claim with suspicion. Rankings shift with each model release and each benchmark, and the differences between frontier models are often smaller than the difference between an easy task and a hard one. The safe posture: verify the output regardless of which model produced it.

The numbers, dated and sourced

Below are the most recent public figures we could verify. They are task-specific and time-stamped on purpose; do not read them as a model's universal error rate.

Two caveats. First, on Vectara's older, easier 1,000-document dataset leading models sat much lower (the best near 1%, most within roughly 1–5%), so the same models look very different across datasets; the task drives the number more than the model does. Second, the Stanford GPT-4 figure is legal-domain and over a year old, included here only to show how high rates climb on hard, high-stakes tasks (Stanford RegLab, 2024). We are not aware of a directly comparable dated Claude figure on the same legal test, so we do not assert one.

Characteristic failure modes

The interesting difference is not a single percentage but how each family tends to fail, which affects what you should check. These are tendencies observed across benchmarks and usage, not laws, and both models exhibit both patterns.

• Confident specifics. When a model lacks a fact, it may still produce a precise-looking name, date, or citation. This is the failure behind fabricated legal cases like Mata v. Avianca, and it is why you should scrutinize any exact figure regardless of the model.
• Misgrounding. A model can cite a real source that does not actually support the claim, the subtle failure the Stanford study highlighted. Reasoning and retrieval features reduce but do not remove it.
• Sycophancy under pressure. Both families can reverse a correct answer if you push back, because they optimize for a satisfying response, not for defending a fact.

The practical upshot: the failure modes overlap enough that a single verification step matters more than picking the "safer" model.

When to trust which, and why to verify either way

A reasonable heuristic, held loosely: for factual summarization of source documents, pick whichever model tops the current Vectara HHEM leaderboard for that task and always ground it in the source text. For open-ended factual questions with no supplied source, treat both as equally capable of confident error and verify every specific claim. Neither model should be relied on unverified for legal, medical, or financial stakes.

Because rankings move, revisit the leaderboard periodically rather than trusting a number you saw once; a model that led last quarter may not lead this one. Whatever you choose, run the output through a verifier. Paste any ChatGPT or Claude answer into /chat, and Mephistopheles grounds each claim against independent sources and returns a per-claim verdict plus a risk score. See the model-specific pages on ChatGPT hallucinations and Claude hallucinations.