Mechanistic Interpretability
Reverse-engineering the computation a trained network learned into human-legible features and circuits.
Mechanistic interpretability is the attempt to reverse-engineer what a trained neural network actually computes, described as features and circuits, rather than treating it as a black box characterized only by its inputs and outputs. A feature is a direction in activation space corresponding to something describable; a circuit is a subgraph of features and weights implementing a behavior end to end. A published example is the induction head, which detects a repeated prefix and copies whatever followed it last time.
The central obstacle is superposition. A model represents far more distinguishable features than it has dimensions, packing them into near-orthogonal directions and tolerating the interference. Individual neurons are therefore polysemantic — one unit fires for several unrelated things — so reading neurons directly yields no explanation. Sparse autoencoders attack this by training a very wide, sparsely activating dictionary on a layer's activations, recovering features closer to monosemantic. Clamping one and watching the output change is the causal test separating a feature from a correlation.
The limits are real. Recovered dictionaries are incomplete, reconstruction is lossy, feature sets differ between training runs, and most fully traced circuits come from small models. Nondeterminism in training and sampling makes exact replication awkward, and a Token-level story about one prompt rarely generalizes.
What none of this licenses is folk prompting advice. Discovering that a Large Language Model carries a feature for a concept says nothing about which phrasing recruits it, and no published result yet justifies an incantation. Claims about Prompt Engineering still need an Evaluation Harness and measured outcomes; interpretability supplies an explanation to be earned, not a Plausible Mechanism to be assumed. Its firmest practical contribution so far is negative — showing that intuitive stories about why a model produced an output, including many about Hallucination, are simply wrong.
See also3
Context Window
The bounded span of tokens a model can attend to in a single request.
Agents & Language Models27 connections
Falsifiability
A claim is only worth something if you know what observation would refute it.
Method47 connections
Markov Chain
A process whose next state depends only on the present state, and the basis of pre-neural text generation.
Agents & Language Models12 connections
Linked from6
- Ablation StudyAgents & Language Models
An experiment that removes or disables one part of a system and measures what changes, borrowed from neuroscience lesion studies and now the standard way machine learning papers show which of their components actually matter.
- Chain-of-Thought PromptingAgents & Language Models
Prompting a model to write out intermediate reasoning steps before its final answer, which measurably improves accuracy on multi-step problems.
- In-Context LearningAgents & Language Models
A language model's ability to pick up a task from examples or instructions in its prompt, without any change to its weights, and the quirks of how it does so.
- Markov ChainAgents & Language Models
A process whose next state depends only on the present state, and the basis of pre-neural text generation.
- Mixture of ExpertsAgents & Language Models
A model architecture that holds many parameter blocks but runs only a few per token, decoupling total size from per-token compute.
- Reasoning ModelAgents & Language Models
Language models trained to think at length before answering, trading tokens and latency for accuracy on hard problems, and the second axis of scaling after model size.