Token
The sub-word unit a language model actually reads and writes; the unit of cost and of context.
A token is the atomic unit a Large Language Model processes. Tokens are not words or characters but fragments produced by a learned segmentation — common words are usually a single token, rare words split into several, and whitespace and punctuation carry tokens of their own. English text averages roughly four characters per token.
Tokens matter for three practical reasons. They are the unit in which the Context Window is measured, so "how much can I show the model" is a token question. They are the unit of billing, so they are the unit of Token Budget. And they are the unit of latency, since generation is sequential.
Tokenization also explains a family of otherwise-strange model behaviors: difficulty with character-level tasks like counting letters or reversing strings, uneven handling of unusual formatting, and the fact that the same content costs different amounts depending on how it is written. Dense structured formats and long identifier names are more expensive than their information content suggests.
Because tokens are consumed by everything in the request — instructions, tool definitions, prior turns, retrieved documents — reducing any of these frees capacity for the rest. That trade is the whole subject of context engineering.
See also7
Context Window
The bounded span of tokens a model can attend to in a single request.
Agents & Language Models20 connections
Token Budget
The finite allowance of model usage a task may consume, and the design decisions it forces.
Agents & Language Models15 connections
Character Encoding
The mapping between characters and bytes, and the source of a specific family of invisible bugs.
Systems & Tooling14 connections
Prompt
The text supplied to a language model to elicit a response.
Agents & Language Models13 connections
Nondeterminism
The property that identical inputs may produce different outputs, and what it costs to test around.
Agents & Language Models23 connections
Tool Use
Giving a model a set of callable functions so it can act on the world rather than only describe it.
Agents & Language Models25 connections
Constrained Decoding
Restricting a model's next-token choices to those a formal grammar permits.
Agents & Language Models11 connections
Related2
Nearby in the graph rather than deliberately chosen. Looser, sometimes surprising.
Linked from16
- Agentic LoopAgents & Language Models
The cycle of model turn, tool call, observation, repeat — the basic mechanism of an AI agent.
- Automatic Speech RecognitionAgents & Language Models
Turning audio into text, and the reasons real-world accuracy is set by the signal more than by the model.
- Character EncodingSystems & Tooling
The mapping between characters and bytes, and the source of a specific family of invisible bugs.
- Constrained DecodingAgents & Language Models
Restricting a model's next-token choices to those a formal grammar permits.
- Context WindowAgents & Language Models
The bounded span of tokens a model can attend to in a single request.
- JSON SchemaAgents & Language Models
A vocabulary for describing the shape of JSON data, and the lingua franca of model tool interfaces.
- Large Language ModelAgents & Language Models
A neural network trained to predict text, used as a general-purpose instruction-following system.
- Markov ChainAgents & Language Models
A process whose next state depends only on the present state, and the basis of pre-neural text generation.
- Mechanistic InterpretabilityAgents & Language Models
Reverse-engineering the computation a trained network learned into human-legible features and circuits.
- Model RoutingAgents & Language Models
Choosing which model handles which step, trading capability against cost and latency.
- NondeterminismAgents & Language Models
The property that identical inputs may produce different outputs, and what it costs to test around.
- PromptAgents & Language Models
The text supplied to a language model to elicit a response.
- Structured OutputAgents & Language Models
Constraining a model to emit data conforming to a declared schema rather than free text.
- Temperature SamplingAgents & Language Models
A parameter that scales how sharply a model favors its most likely next token, trading determinism for variety.
- Token BudgetAgents & Language Models
The finite allowance of model usage a task may consume, and the design decisions it forces.
- Tool UseAgents & Language Models
Giving a model a set of callable functions so it can act on the world rather than only describe it.