Questions

Technology

Will AI always make things up?

Training and evaluation can encourage language models to make up answers rather than admit they do not know, according to a study published in Nature. A preprint presents a proof that no language model can eliminate these fabrications, but the result depends on the paper’s assumptions.

Engraving: a bust of the baron, with a huge moustache and wig, on a pedestal with a coat of arms.
Baron Munchausen, by Gustave Doré (1866), the hero of made-up stories. Public domain. Wikimedia Commons

Why it matters

How AI is evaluated matters: rewarding guesses can favor made-up answers. Errors can also be measured in specific tasks, as Vectara’s leaderboard does by checking whether summaries include something that was not in the original text.

What the evidence shows

  1. A preprint presents a mathematical proof that, under the paper’s assumptions, no language model can eliminate hallucination.Hallucination is Inevitable: An Innate Limitation of Large Language Models · Ziwei Xu, Sanjay Jain, Mohan Kankanhalli, arXiv, 2024
  2. A study published in Nature argues that training and evaluation reward guessing, making a fabricated answer score better than admitting uncertainty.Evaluating large language models for accuracy incentivizes hallucinations · Adam Tauman Kalai et al., Nature 653, 1047–1051, 2026
  3. Vectara’s leaderboard applies the same test to every model to measure the share of summaries containing made-up information absent from the original text.Hallucination Leaderboard (HHEM-2.3) · Vectara, atualizado em 22 out. 2026, 2026

What I would do

I recommend, as you expand your use of AI, measuring errors, checking sources, and requiring it to admit when it does not know and refer the question to a person.

Why I think so:

  • “AI without evaluation is a bet: measure before you promise.” where it comes from
  • “You can trust AI, if you check it.”
  • “A serious AI says only what it knows, shows where it got it, says when it does not know, and calls a person.” where it comes from

People also ask

  • Why does AI hallucinate?
  • Can you trust what AI tells you?

Read next