Token
The sub-word unit a language model actually reads and writes; the unit of cost and of context.
A token is the atomic unit a Large Language Model processes. Tokens are not words or characters but fragments produced by a learned segmentation — common words are usually a single token, rare words split into several, and whitespace and punctuation carry tokens of their own. English text averages roughly four characters per token.
Tokens matter for three practical reasons. They are the unit in which the Context Window is measured, so "how much can I show the model" is a token question. They are the unit of billing, so they are the unit of Token Budget. And they are the unit of latency, since generation is sequential.
Tokenization also explains a family of otherwise-strange model behaviors: difficulty with character-level tasks like counting letters or reversing strings, uneven handling of unusual formatting, and the fact that the same content costs different amounts depending on how it is written. Dense structured formats and long identifier names are more expensive than their information content suggests.
Because tokens are consumed by everything in the request — instructions, tool definitions, prior turns, retrieved documents — reducing any of these frees capacity for the rest. That trade is the whole subject of context engineering.
See also10
Prompt
The text supplied to a language model to elicit a response.
Agents & Language Models14 connections
Constrained Decoding
Restricting a model's next-token choices to those a formal grammar permits.
Agents & Language Models11 connections
Prompt Engineering
Shaping a model's input to make the desired behavior reliable rather than occasional.
Agents & Language Models22 connections
Structured Output
Constraining a model to emit data conforming to a declared schema rather than free text.
Agents & Language Models13 connections
Character Encoding
The mapping between characters and bytes, and the source of a specific family of invisible bugs.
Systems & Tooling31 connections
Nondeterminism
The property that identical inputs may produce different outputs, and what it costs to test around.
Agents & Language Models24 connections
Tool Use
Giving a model a set of callable functions so it can act on the world rather than only describe it.
Agents & Language Models27 connections
Stemming and Lemmatization
Reducing words to a common base form so a search for "running" also finds "runs" and "ran," with stemming chopping suffixes by rule and lemmatization looking up the dictionary form.
Data & Content7 connections
Lexical Analysis
The first stage of reading a program, which groups a stream of characters into tokens such as names, numbers, operators and keywords, usually using regular expressions compiled into a finite automaton, so the parser can work with words rather than letters.
Computation & Algorithms9 connections
Byte-Pair Encoding
The compression trick that splits text into the tokens a language model reads, by repeatedly merging the most frequent pair of symbols into a new one.
Agents & Language Models8 connections
Linked from20
- Agentic LoopAgents & Language Models
The cycle of model turn, tool call, observation, repeat — the basic mechanism of an AI agent.
- Automatic Speech RecognitionAgents & Language Models
Turning audio into text, and the reasons real-world accuracy is set by the signal more than by the model.
- Byte-Pair EncodingAgents & Language Models
The compression trick that splits text into the tokens a language model reads, by repeatedly merging the most frequent pair of symbols into a new one.
- Character EncodingSystems & Tooling
The mapping between characters and bytes, and the source of a specific family of invisible bugs.
- Constrained DecodingAgents & Language Models
Restricting a model's next-token choices to those a formal grammar permits.
- Context WindowAgents & Language Models
The bounded span of tokens a model can attend to in a single request.
- JSON SchemaAgents & Language Models
A vocabulary for describing the shape of JSON data, and the lingua franca of model tool interfaces.
- KV CacheAgents & Language Models
The stored attention keys and values for every token already processed, which lets a model generate each new token without re-reading the whole sequence.
- Large Language ModelAgents & Language Models
A neural network trained to predict text, used as a general-purpose instruction-following system.
- Lexical AnalysisComputation & Algorithms
The first stage of reading a program, which groups a stream of characters into tokens such as names, numbers, operators and keywords, usually using regular expressions compiled into a finite automaton, so the parser can work with words rather than letters.
- Markov ChainAgents & Language Models
A process whose next state depends only on the present state, and the basis of pre-neural text generation.
- Mechanistic InterpretabilityAgents & Language Models
Reverse-engineering the computation a trained network learned into human-legible features and circuits.
- Mixture of ExpertsAgents & Language Models
A model architecture that holds many parameter blocks but runs only a few per token, decoupling total size from per-token compute.
- Model RoutingAgents & Language Models
Choosing which model handles which step, trading capability against cost and latency.
- NondeterminismAgents & Language Models
The property that identical inputs may produce different outputs, and what it costs to test around.
- PromptAgents & Language Models
The text supplied to a language model to elicit a response.
- Structured OutputAgents & Language Models
Constraining a model to emit data conforming to a declared schema rather than free text.
- Temperature SamplingAgents & Language Models
A parameter that scales how sharply a model favors its most likely next token, trading determinism for variety.
- Token BudgetAgents & Language Models
The finite allowance of model usage a task may consume, and the design decisions it forces.
- Tool UseAgents & Language Models
Giving a model a set of callable functions so it can act on the world rather than only describe it.