Why AI Makes Things Up: 7 Checks That Catch a Hallucination

AI makes things up because a language model is built to produce the most likely next words, not to check whether they are true, and because the way models are trained and scored rewards a confident guess over an honest "I don't know". This guide explains the mechanism in plain English, how often it happens on current models, what it cost three firms and the seven checks that catch it.

You have seen it: the chatbot that cites a regulation that does not exist, quotes a statistic no survey ever produced, or names a supplier you cannot find. It is called a hallucination, and it is not a bug that the next update will remove. OpenAI's own researchers wrote in September 2025 that models produce "overconfident, plausible falsehoods" because their training and evaluation "reward guessing over acknowledging uncertainty". Understanding why AI makes things up is the difference between a tool that saves you a day a week and one that puts a fake figure in front of a customer, a bank or a court.

What follows is the explanation a busy owner needs, not a computer science lecture: how the machine actually works, where the errors cluster, three real cases with the bill attached, what the vendors themselves tell you, and the seven checks. The last section covers how an AI agent that shows its sources and its browser changes the problem, and what it still cannot do for you.

How we checked this guide

1. Primary sources only for the mechanism: the OpenAI research paper of 4 September 2025 and the regulator statements that quote the vendors.
2. Measured rates from a public leaderboard read on 22 October 2026, with its method stated, because a hallucination rate without a method is itself a kind of hallucination.
3. Court and government records for the three cases, not news summaries.
4. The vendors' own help pages for what ChatGPT and Claude say about their limits.
5. Checks that a non-technical person can run in under a minute, without any tool but a search box.

What a hallucination is, and why it happens

A large language model works by predicting the text most likely to follow the text it has been given. When the Solicitors Regulation Authority warned the legal profession in November 2023, it put it plainly: these tools "work by anticipating the text that should follow the input they are given, but do not have a concept of 'reality'". Ask for a case that supports your argument and the model produces something case-shaped: a plausible name, a plausible year, a plausible quotation. Nothing inside it checks whether the case exists, because nothing inside it was built to.

The second cause is how models are scored. In their September 2025 paper "Why language models hallucinate", four OpenAI researchers showed that the standard tests reward a confident answer and give nothing for admitting uncertainty, so "under binary grading, abstaining is strictly sub-optimal". A model that guesses gets more points than a model that says it does not know, in the same way a student who guesses on a multiple-choice exam beats one who leaves the question blank. The guessing habit is learned, and it is learned because it wins.

The figures OpenAI published alongside the paper make the trade-off visible. On one factual benchmark, a model tuned to abstain declined to answer 52% of questions, got 22% right and 26% wrong; an earlier model that almost never abstained got 24% right and 75% wrong. Fewer wrong answers came from more "I don't know", not from more knowledge. That is the whole story in one table, and it is why the checks below start with permission to say no.

How often it happens in 2026

The honest answer is "often enough to check, always". Vectara's hallucination leaderboard, updated on 22 October 2026, scores models on one narrow task: summarise a document they have been given, and count how often the summary contains something the document does not say. Even with the facts in front of them, mainstream models invented material in a measurable share of summaries.

Two things stand out. Newer and bigger does not mean safer: the August 2025 GPT-5 configuration scored worse than the year-older GPT-4o on this test. And these are grounding figures, with the source supplied; when the model has to recall facts from memory, the rates are higher.

The specialist studies agree. Researchers writing in the Annals of the Royal College of Surgeons of England reported in April 2026 that across 108 chatbot answers to surgical questions, containing 1,249 references, the worst models fabricated or could not support 25% to 34% of their citations, while the best produced none. A Lancet letter reported in May 2026 found that about one in 277 recent papers in PubMed carried fabricated references, against one in 2,828 in 2023. A Stanford study of legal research tools that use retrieval found they still hallucinated between 17% and 33% of the time. Retrieval helps; it does not finish the job.

What it has cost: three cases

New York, June 2023. In Mata v Avianca, two lawyers filed a brief containing cases that did not exist, generated by ChatGPT, and the court imposed a $5,000 penalty. It was the first widely reported case, and it is now one of more than 2,200 court decisions worldwide involving fabricated AI citations tracked in a public database maintained by the lawyer Damien Charlotin.

London, June 2025. In Ayinde v Haringey and Al-Haroun v Qatar National Bank, the High Court dealt with two cases at once: one brief cited five cases that did not exist, the other cited 45 authorities of which 18 were invented. The President of the King's Bench Division wrote that generative tools "are not capable of conducting reliable legal research" and "may cite sources that do not exist", that lawyers have a professional duty "to check the accuracy of such research by reference to authoritative sources", and that the consequences of failing to do so can include contempt proceedings and referral to the police. Since then the Solicitors Regulation Authority has issued a warning notice (17 August 2026) naming "fictitious cases, references or seemingly factual assertions" and reminding solicitors that they remain accountable for the output.

Canberra, October 2025. Deloitte Australia delivered a report to a government department under a contract worth AU$440,000. It contained a fabricated quotation attributed to a court judgment and references to academic papers that did not exist. The firm published a corrected version on 3 October 2025, disclosed for the first time that an Azure OpenAI model had been used, and repaid the final instalment of roughly AU$98,000. The work was done by a large professional firm with a review process. The fabrication still got through, because nobody opened the references.

None of these people were careless amateurs. They were busy professionals who treated fluent output as checked output. That is the mistake to avoid, and it is avoidable.

What the vendors say themselves

Anthropic's documentation for Claude lists the standard mitigations, including allowing the model to say "I don't know", asking it to quote before it answers and restricting it to supplied documents, and then adds that these "don't eliminate them entirely. Always validate critical information." Its support article on incorrect answers, updated in March 2026, says plainly that Claude "can occasionally produce responses that are incorrect or misleading" and should not be relied on as "a singular source of truth". OpenAI's help page on whether ChatGPT tells the truth says the same: it can fabricate citations, and you should verify. The ICO, in its 2024 work on generative AI, noted that even accurate training data "will not stop" models producing false statements, and the FCA's October 2026 review of AI in financial services repeats that firms "remain responsible for consumer outcomes" whatever tool produced them.

Read together, the message from the people who build and regulate these tools is consistent: use them, and check them.

The 7 checks that catch a hallucination

1. Ask for the source, then open it

A model will happily produce a reference on request; the test is whether the reference opens and says what the model says it says. The Deloitte report had references. Nobody opened them. If the link is dead, the paper does not exist or the page says something different, the whole answer is suspect, not just that line.

2. Verify names, numbers, dates and quotes individually

These four categories are where fabrication concentrates, because they are the parts of a sentence that are hard to predict from context. A quotation is the easiest to check: paste it into a search engine inside quotation marks. A statistic should lead to a survey with a sample and a date, in the same way this guide names the Royal College of Surgeons study and its 1,249 references.

3. Ask twice, or ask a second model

A fact that survives rephrasing is more likely to be a fact. Ask the same question with different wording, or paste it into a second product (ChatGPT, Claude, Gemini, Microsoft Copilot, Manus or Meta's Muse all behave differently on the same prompt). Two models agreeing is not proof, but two models disagreeing is a reliable signal to check.

4. Give it the document and fence it in

The leaderboard figures above are the best case, with the source supplied; recall from memory is worse. So supply the source. Attach the contract, the spreadsheet or the regulation, tell the model to answer only from it and to quote the line it used, and read the quote. If it cannot quote a line, the answer did not come from the document.

5. Give it permission to say "I don't know"

Because the training rewards guessing, you have to change the scoring in the prompt. "If you are not sure, say so and tell me what you would need to check" is the single most effective sentence in this guide; it is the written equivalent of the abstaining model in OpenAI's table, and it costs you nothing.

6. Redo the maths

Language models predict text; they do not compute unless they are running code. A total that is off by one line, a percentage of the wrong base, a date that is a year out: these look right because the surrounding sentence is right. Add the column yourself, or ask an agent that runs the calculation in a spreadsheet and shows it.

7. Treat money, law, health and reputation as draft

The lawyers in London and the consultants in Canberra did not lack intelligence; they lacked a rule. The rule is that anything which reaches a client, a regulator, a lender or a court is a draft until a named person has checked the facts in it. Write that rule down, and the AI policy in AI Policy for Small Business: A Free Template gives you the one-page version.

Where small businesses get caught most

• Quotes and estimates. A model asked for "typical prices" invents typical prices. Use your own rate card or a priced source.
• Legal and tax wording. Thresholds, dates and penalties change every April; the model's memory may be two years old. Check GOV.UK.
• Supplier and product details. Model numbers, compatibility and specifications are exactly the dense, hard-to-predict text that gets fabricated.
• Reviews and testimonials. Never let a model write a review or a quote from a customer. Beyond being invented, fake reviews are illegal in the UK under the Digital Markets, Competition and Consumers Act 2024.
• Translations of figures. When a model translates a document, numbers can shift. Check every figure in the translated version against the original.

How an AI agent changes the problem

A chatbot gives you an answer and leaves the checking to you. An agent can do some of the checking as part of the job, which is why the mechanics matter when you choose one.

• It shows its sources. An agent that browses returns the pages it read, so check 1 becomes a click rather than a search. Jobbit Research attaches the sources to every finding in the report it writes.
• It runs in a browser you can watch. When the agent opens a page, you see the page. A figure in the report can be traced to the screen it came from.
• It works from your documents. Attach the contract or the spreadsheet in a Jobbit project and the agent answers from the file, with the line quoted, which is check 4 done by default.
• It does the maths in a spreadsheet, not in prose. Ask for a cash flow forecast and the agent builds the sheet and shows the formulas; the arithmetic is reproducible.
• It asks before it spends or sends. An agent that pauses for approval before an email goes out gives check 7 a built-in stop.
• It can be told to abstain. Project instructions such as "say when you are not sure and list what to verify" apply to every chat, so check 5 is set once.

ChatGPT, Claude, Gemini, Microsoft Copilot, Manus and Meta's Muse all draft well, and several now browse and cite. What none of them removes is the final check by a person, and neither does Jobbit. What Jobbit adds is a human network: when the job is a lease, a tax return or a medical claim, the agent can hand the checking to a vetted professional inside the same order rather than to you at eleven at night.

Your plan for this week

• Monday: write the one rule. "Anything that goes to a client, HMRC, a bank or a court is a draft until a named person has checked the facts." Put it in your AI policy and your project instructions.
• Tuesday: add the abstain line. Put "if you are not sure, say so and tell me what to check" into the standing instructions of every tool you use.
• Wednesday: run check 1 on last week's output. Take the last three AI-written documents you used and open every reference. Count the ones that fail.
• Thursday: move one job to grounded mode. Pick the task where errors cost most, attach the real documents and require quoted lines.
• Friday: decide what never goes out unchecked. Make the list (prices, legal wording, health claims, customer quotes) and share it with anyone who uses AI in your business.

Read next

• Is AI Safe to Use? A Plain-English Guide to Your Data, Your Money and the Scams to Watch
• 12 Mistakes Beginners Make With AI Agents, and the Quick Fix for Each One
• How Do AI Agents Work? The 5-Step Loop Behind Every Task, Explained Simply

Frequently asked questions

Why does AI make things up?

Because a language model generates the most probable next words rather than checking facts, and because the tests used to train and rank models reward a confident guess over an admission of uncertainty. OpenAI's own researchers described the result in 2025 as "overconfident, plausible falsehoods". Fluency and accuracy are produced by the same process, which is why a wrong answer reads exactly like a right one.

What is an AI hallucination?

An answer that is fluent and plausible but false: an invented case, a statistic that no survey produced, a quotation nobody said, a product that does not exist. It is not a glitch; it is the normal behaviour of a model asked for something it does not know.

How often does ChatGPT make things up?

It depends on the task. On a public grounding test read in October 2026, mainstream models invented material in roughly 4% to 15% of document summaries even with the source supplied; studies of citations in surgery and law found fabrication rates of 17% to 34% for the worst performers. The safe assumption is "often enough to check every name, number, date and quote".

Can you stop AI hallucinations completely?

No. Supplying the source document, asking for quotes, allowing "I don't know" and using an agent that shows its sources all reduce the rate, and Anthropic's own documentation says these measures "don't eliminate them entirely". The final check on anything that matters is a person.

Has anyone been fined for AI hallucinations?

Yes. A New York court imposed a $5,000 penalty in 2023 for fake cases in a brief; the High Court in London warned in June 2025 that fabricated citations can lead to contempt proceedings and referral to the police; and Deloitte Australia repaid the final instalment of a AU$440,000 government contract in October 2025 after its report was found to contain invented references.

Is an AI agent more reliable than a chatbot?

An agent that browses, cites its sources and works from your documents makes the checks faster and the errors easier to trace, but it still needs a person to sign off anything that affects money, law or health. Jobbit's agent shows its browser and its sources as it works, and can pass the final check to a vetted professional when the job needs one.