Record and Replay Testing
Capturing a real interaction once and replaying it deterministically in later test runs.
Record and replay testing captures the real exchange with an external service — the request and its response — into a fixture, then serves that fixture on subsequent runs. The pattern is old, and its usual name in the HTTP world is the cassette, after the library that popularised it. It is what makes a suite that depends on a paid, slow, non-deterministic service fast, free, and repeatable.
Applied to Large Language Model systems it has particular properties.
The recording is only valid for the request that produced it. That request includes the system prompt, the conversation, and every tool definition in scope — so an unrelated tool's description being reworded invalidates every recording, even ones that never used it. See Fingerprint.
Failures deserve recording too. If only successful calls are captured, a path that recovers from a rejection has nothing to replay and behaves differently under test than in production. Recording the rejection lets replay reproduce the recovery.
Re-recording has collateral. Tests elsewhere that pin values derived from a recording — a token count, a specific sentence, an opaque identifier — go red, and read as unrelated breakage mid-session.
Recordings are also a corpus meant to be read later, so a false verdict inside one is worse than a gap. See Provenance.
See also5
Hand-picked in the note itself — the neighbours worth reading next.
Evaluation Harness
A repeatable test suite for model behaviour, since prompts and tool descriptions cannot be type-checked.
Agents & Language Models11 connections
Test Fixture
The prepared data or state a test runs against.
Testing & Verification12 connections
Determinism
The property that identical inputs produce identical outputs, and the foundation of caching and verification.
Systems & Tooling27 connections
Test Double
A stand-in for a real dependency during a test — and the ways a stand-in can lie.
Testing & Verification10 connections
Nondeterminism
The property that identical inputs may produce different outputs, and what it costs to test around.
Agents & Language Models17 connections
Related2
Nearby in the graph rather than deliberately chosen. Looser, sometimes surprising.
Linked from9
Notes elsewhere in the wiki that reach for this one.
- Cache InvalidationWeb Platform
Deciding when a cached value has stopped being correct — famously one of the hard problems.
- DeterminismSystems & Tooling
The property that identical inputs produce identical outputs, and the foundation of caching and verification.
- Evaluation HarnessAgents & Language Models
A repeatable test suite for model behaviour, since prompts and tool descriptions cannot be type-checked.
- FingerprintAgents & Language Models
A hash over everything that determined a result, used to detect when the result has gone stale.
- Hash FunctionSystems & Tooling
A function mapping arbitrary input to a fixed-size digest, used for identity, integrity, and addressing.
- LLM-as-JudgeAgents & Language Models
Using a language model to score another model's output against a rubric.
- NondeterminismAgents & Language Models
The property that identical inputs may produce different outputs, and what it costs to test around.
- Test DoubleTesting & Verification
A stand-in for a real dependency during a test — and the ways a stand-in can lie.
- Test FixtureTesting & Verification
The prepared data or state a test runs against.