What AI hallucination is

Hallucination is a model stating something in exactly the tone it uses for the truth, when that thing does not exist at all. It is not the model breaking: the model builds the most probable continuation of the text, and a fake source that looks right is, statistically, just as probable as a real one.

  • Lesson 12 of 15
  • Beginner
  • Free, no signup
Working with AI

What you read, and what is not under it

Above the line, a sentence in which everything looks right. Below it, the things that sentence claims exist.

A fluent, confident answer

The same tone it uses for a correct answer. Tone carries no information about correctness.

the surface you actually read
  • A source that does not exist

    Author, year and journal, all in the right shape. Searching the title answers it in seconds.

  • A number that is too well shaped

    Round, tidy and exactly the size you expected. A real number is usually uglier.

  • A rule or clause that was made up

    A statute number, a policy clause, a setting name, a function name. The shape is predictable, so it gets built.

  • A real source, for a claim it never makes

    It survives a link check. The only way through is to find the sentence on the page.

None of these four shapes is rare, and none of them leaves a mark on the sentence. The only way to see them is to go to the source itself.

Last checked: Facts and tool names in this lesson are re-checked against their sources on this date.

Why the word "hallucination" misleads

Hallucination is the name given to this: a model producing a sentence in which everything looks right and nothing is actually backed by anything. A paper with an author, a year and a journal that was never written. A statute with a number that does not exist. A figure that appears in no source.

The trouble is the name itself. "Hallucination" suggests something broke and the model left its normal state. What actually happens is the reverse. In that moment the model is doing precisely what it does when it is right: building the most probable continuation. When the answer is correct, the most probable continuation happens to line up with reality. When it is wrong, it does not. From inside that calculation the two cases are not distinguishable at all.

Our position is that this distinction is practical, not philosophical. Treat hallucination as a defect and you wait for the next release to fix it. Treat it as ordinary behaviour of the same mechanism and, instead of waiting, you arrange your process so that every checkable claim gets checked. The second one works.

An iceberg of white blocks above water dissolving into blue mist and hollow red outlines below a green line

Why prediction produces confidence

A language model builds text token by token, choosing the most probable continuation at each step. The mechanism is set out in the lesson on what a large language model is. What matters here is one line from it: there is no input wire called source of truth connected to this calculation.

So what happens when you ask a model for a source. The model has learned the shape of a citation extremely well: a surname, a year in brackets, a title that fits the subject, the name of a journal that carries weight in that field. All of those are high-frequency patterns in text. What it has not learned is whether this particular citation exists, because that fact has no place to sit in the calculation.

And then the confident tone. The model is optimised for fluency, and fluency means a sentence without hedging. A model that says perhaps and I am not sure all the time scores worse in evaluations, so these tools have been pushed towards decisiveness. The consequence for you is simple and unpleasant: the model tone carries no information about whether the answer is right. The wrong answer and the right one arrive with identical confidence, and that is what makes this dangerous.

Learn the classic failure shapes by heart

Hallucination does not take random shapes. It has a handful of specific patterns, and once you know them you catch most of them on the first read.

  • A source that does not exist. The most common shape and the most expensive. A paper, a book, a report or a documentation page with every detail right and no existence.
  • A number that is too well shaped. Invented figures tend to be round and tidy and to land exactly where you expected them. Real numbers are usually uglier.
  • A rule or clause that was made up. A statute number, a policy clause, the name of a setting in a piece of software, the name of a function in a library. These have highly predictable shapes, and the model builds the shape.
  • A real source, for a claim it never makes. The worst of the four, because it survives a link check. The link opens, the page is genuine, and the sentence attributed to it is not on it.

That last one is where most editorial teams come apart. Opening the link is not enough; you have to find the sentence on the page. We apply the same rule to ourselves, and there is an instance of it written up in the lesson on detecting AI-generated content: a number we could not read at its source did not get printed.

How to check the answer

Several techniques genuinely work here, and the vendors themselves have written down which ones. Anthropic documentation lists four basics: explicitly allow the model to say it does not know, for long documents ask for word-for-word quotes first and analysis second, make every claim find a supporting quote and retract the claim if it cannot, and tell the model to work only from the documents given and add nothing from its own knowledge.

And on that same page, the sentence that matters more than all four: while these techniques significantly reduce hallucinations, they do not eliminate them entirely, and critical information should always be validated, especially for high-stakes decisions. That is the model maker writing, not a critic.

There is one more technique the same documentation points at, and it is very fast in practice: run the same prompt several times and compare the outputs; inconsistencies across them can indicate hallucination. What we have added from our own work is to do this in fresh conversations rather than the same one, or the model sees its own earlier answer and falls in line with it.

And the boundary all of these techniques run into: a model checking its own answer is not an independent check. If a claim came from nowhere in the first place, asking the same model for a supporting quote can add a supporting quote to it rather than catch it.

Four steps that make a claim defensible

The order matters. The first three steps can be handed to a model and the fourth cannot.

  1. Give permission not to know

    Write it out: if you are not sure, say you do not know. Without that sentence the model builds something.

    1
  2. Quotes first, analysis second

    For a long document, ask for word-for-word sentences first and build the analysis on those.

    2
  3. Retract any claim with no quote

    Every claim has to find a supporting quote, and if it cannot, it comes out on the spot.

    3
  4. Open the source yourself

    The link opening is not enough; the sentence has to be on the page. This step cannot be delegated.

    4

None of these four steps takes hallucination to zero. The model maker writes the same thing.

Where you should not be asking a model at all

The rule is simple and has nothing to do with the subject: if you cannot check the answer, do not ask. The right question is not whether the model is good in this field; it is whether you would notice if it answered wrongly.

Some specific places where the cost of a mistake is not yours to bear: law and contracts, medicine and dosage, prices and rates and fees, dates and deadlines, and any sentence about a real person or a real company. In those, a plausible wrong sentence is worse than an obviously wrong one, because it passes without resistance.

And a point that matters in the Iranian market in particular. Anything local, meaning Iranian law, domestic prices, the names and terms of Iranian companies, is exactly where these models have seen the least text. Seeing less does not stop them answering. They answer, with the same confidence, and because the question is local you probably cannot check it quickly either. That combination is the worst case there is.

Where to ask and where not to

The dividing line is not the subject. It is whether you would find out, within minutes, that the answer was wrong.

Where you can check the answer

  • Summarising a text you supplied yourself
  • Rewriting and editing something you wrote
  • Code you run and whose errors you see
  • Generating options when you do the choosing

Where the cost of a mistake is not yours

  • Law, contracts and anything with clause numbers
  • Medicine, drugs and dosage
  • Prices, rates and fees
  • Any sentence about a real person or company

The second column does not mean the model is weak on those subjects. It means somebody else pays for the mistake.

The rule we hold ourselves to

The editorial policy of this site states that data, numbers and technical claims are checked before publication and reconciled with a source where necessary, and that we use AI tools to help with research and editing but publish nothing without review and approval by a human specialist.

But a sentence on a policy page is only a promise. What turns it into a rule is taking the enforcement out of human hands. So in this Learn section, any figure that displays a number must carry, next to that number, a string that the lesson body also prints; otherwise the validator rejects the file and the lesson does not come up at all. The detail is in the experience block further down this page.

And honestly: the rule has a cost. More than once we have had a number that fitted the lesson, could not read it at its source, dropped it, and wrote a sentence about why instead. The page looks weaker for it. But a reader who trusts one invented number of yours and later finds out does not come back, and that cost is much larger.

The fast path, with AI

The fast path here is not to hunt for errors after you have the answer. What a professional does is <strong>reverse the order</strong>: sources first, answer second. When a model has to say, before writing any claim, where it comes from and whether it can quote the sentence, a large part of the inventing simply does not happen. A fast, cheap model is enough for the first pass and a stronger one is needed for the final judgement; our current picks are set out in the <a class="text-link" href="/en/ai/">AI section</a>.

  1. Before any answer, ask for the list of sources, and for each one ask whether it can quote a sentence word for word.
  2. Ask for the answer only from that list, and say explicitly that anything unsupported must be marked as such.
  3. Ask the same question again in a completely fresh conversation and put the two answers side by side. Wherever they differ is the weak point.
  4. Open the links yourself and find the sentence on the page. Do not hand this step to any model.

Copy-ready recipe

== Pass 1: sources before the answer (fast, cheap model)

You are going to answer me about {topic}, but do not answer yet.
First give a table with three columns and write nothing else:

  the claim you are about to make | the source you rely on | can you
  quote a sentence from that source word for word? yes or no

Hard rules:
- If the third column is no, mark that row "no source".
- Do not build a source from its shape. If you are not sure it exists, leave it out.
- If you have no source at all for this subject, write only: I have no source.


== Pass 2: the answer, from that table only (stronger model)

Now write the answer and use only the rows whose third column was yes.
Wrap any sentence resting on a "no source" row in [no source].
Add nothing from your own general knowledge, even if you are sure it is right.


== Pass 3: the diff (fresh conversation, no history)

Open a completely new chat and ask exactly the question from pass 1.
Put the two tables side by side. Any row present in one and absent from the
other, or carrying a different source, is where you go and check yourself.


== Pass 4: no model

Open every link and find the sentence on the page.
The link opening proves nothing.

Before you trust the output: This recipe reduces the number of hallucinations and does not take it to zero, and the model maker writes the same thing. Know three things. First, the model can invent a source in pass one too; the table only focuses your work and shows you where to check. Second, two answers agreeing in pass three is not proof: a fake source built on a high-frequency pattern can repeat in both conversations. Third, and above both, pass four cannot be dropped. If you do not have time to open the links, you do not have time to publish the piece.

AI in this kind of work

Our position here is that the answer is not a better model. The answer is a workflow that hands you the result together with something you can open. A tool that only answers, however strong, leaves you exactly where you started. A tool that ties the answer back to a source you supplied yourself genuinely changes the work.

Tools that actually help

  • NotebookLM Our first pick for this problem, because its shape is right: it answers only from sources you upload yourself and points every sentence back at the source. That does not remove hallucination, but it turns checking from a slog into a click. Google own help says it works in the same regions as the Gemini app, and Iran is not on that list.
  • Gemini Google own documentation describes grounding, which connects a model answer to search results and returns citations. Note that this is not automatic verification either; it just gives you something you can open. Iran is not on the supported-countries list for the Gemini app; we read that on Google own page rather than measuring it.
  • Claude Their own documentation has a page listing the techniques for reducing hallucination, and states on the same page that these techniques do not eliminate it. It suits the first pass of the recipe above, because it follows the instruction to work only from the document supplied. Iran is on neither of Anthropic two supported-countries lists.

Where it backfires

Three risks specific to this topic. First, asking a model for sources can itself produce hallucination: the model knows the shape of a citation and, absent a real one, builds the shape; so a list of sources without opening the links is false security. Second, a model checking its own answer is not independent and can add a justification to a baseless claim rather than catch it. Third, attributing a sentence to a real source that never made it survives a link check, which is what makes it the most dangerous shape; the way through is to find the sentence itself on that page. And one more warning that has nothing to do with accuracy: a client unpublished draft is their text, and whether a conversation is used for model training depends on the plan and settings of that service and is written on their own page.

Sources: Anthropic: reduce hallucinations Google: grounding in the Gemini API Google: where the Gemini apps are available

Where this advice stops

This lesson prints no hallucination rate for any model, because we have no defensible source for such a number and because any figure you see anywhere goes stale fast. Comparing models on this is not our work either. The practical boundary of this lesson lies elsewhere: every method here assumes you can check the answer. If you do not know a subject at all and cannot tell a good source from a bad one, no workflow will save you; there the answer is to ask somebody who knows. And a final caveat: hallucination is only one way to be wrong. A model can produce a sentence in which every word is true and the whole conclusion is false, and this lesson is not about that.

From our own work

In this section we have taken the rule "no number without a source gets printed" out of human hands. The validator of this Learn section, in the plugin file includes/registry.php, holds a list of chart types, and every numeric item in those types must carry both a value and a display string; if either is missing the file is rejected and the lesson does not come up at all. That the same string must also be printed in the lesson body is a rule we keep by hand, but the validator is what closes the door on an ownerless number. And a consequence we did not expect: on the day this lesson was written, the data files of this section held two hundred and twenty six figures and only fourteen of them were numeric charts, carrying fifty six numbered items between them. Which means nearly every figure in this section deliberately states no number at all. When a number with no source simply cannot be drawn, what gets drawn instead is a shape that makes no numeric claim, and the page has lost nothing by it.

Real follow-up questions

Do newer models not hallucinate?

Less, but not zero, and we print no comparison number because we have no defensible source for one. More to the point, the mechanism has not changed: the model still builds the most probable continuation and still has no input wire called source of truth. Anthropic own documentation likewise states that the techniques for reducing hallucination do not eliminate it entirely.

If I ask the model for sources, is the problem solved?

No, because the list of sources can be invented as well. The model knows the shape of a citation very well and builds that shape. A source list is worth something only when you open the links and find the sentence on the page; the link merely opening is not enough.

Why does the model not just say it does not know?

Because I do not know is not a probable continuation unless you make it one. The model is optimised for fluency and decisiveness, so it produces sentences without hedging. Write explicitly in the prompt that it should say it does not know when it is unsure and the behaviour changes; vendor documentation offers exactly this as the first technique.