Hallucination
Fluent, confident output that is not grounded in anything real.
Hallucination is the production of plausible, well-formed content that has no basis in fact — a cited source that does not exist, an API method that was never defined, a file path that is invented rather than observed. The term is imperfect (confabulation is closer) but it has stuck.
It is not a bug in the ordinary sense. A Large Language Model generates likely continuations; a fluent wrong answer and a fluent right answer are equally likely-looking from inside the model. Nothing in the objective distinguishes them.
The important consequence is that fluency is not evidence, and specificity is actively misleading: a precise-sounding mechanism reads as more rigorous than an honest "not established", which is exactly the trap described in Plausible Mechanism. Humans make the same error for the same reason, which is why the mitigation is procedural rather than model-specific.
Mitigations that work are all about grounding. Give the model tools to look rather than recall — reading the file beats remembering the file. Validate identifiers against a real catalog rather than trusting recall. Require citations that can be checked. And design so that unverifiable claims are cheap to detect: an exhaustive claim, an invented path, or a cited test file are each one command away from being falsified. See Ground Truth and Exhaustive Claim.
See also4
Falsifiability
A claim is only worth something if you know what observation would refute it.
Method46 connections
Tool Use
Giving a model a set of callable functions so it can act on the world rather than only describe it.
Agents & Language Models25 connections
Guardrail
A rule constraining what a model may do, enforced by prompt, schema, or surrounding code.
Agents & Language Models14 connections
Nondeterminism
The property that identical inputs may produce different outputs, and what it costs to test around.
Agents & Language Models23 connections
Related2
Nearby in the graph rather than deliberately chosen. Looser, sometimes surprising.
Linked from10
- AssertionTesting & Verification
The statement in a test that must hold, and the only part that can actually fail.
- Automatic Speech RecognitionAgents & Language Models
Turning audio into text, and the reasons real-world accuracy is set by the signal more than by the model.
- Automation BiasMethod
Over-trusting an automated recommendation, including against available contrary evidence.
- Exhaustive ClaimMethod
A statement of the form "the last one" or "nothing else does this" — load-bearing, and only as good as the search behind it.
- Large Language ModelAgents & Language Models
A neural network trained to predict text, used as a general-purpose instruction-following system.
- Linguistic RelativityMeaning & Society
The hypothesis that language shapes thought, and the evidence that has narrowed it considerably.
- Markov ChainAgents & Language Models
A process whose next state depends only on the present state, and the basis of pre-neural text generation.
- Mechanistic InterpretabilityAgents & Language Models
Reverse-engineering the computation a trained network learned into human-legible features and circuits.
- Plausible MechanismMethod
A causal explanation that was inferred rather than tested, and reads as more rigorous for being specific.
- Retrieval-Augmented GenerationAgents & Language Models
Fetching relevant documents at query time and feeding them into the prompt, so a model answers from retrieved text rather than memory.