Nondeterminism
The property that identical inputs may produce different outputs, and what it costs to test around.
Nondeterminism in language-model systems means the same request can yield different responses. It follows from sampling: generation draws from a probability distribution over next Tokens, and even at the lowest sampling temperature, floating-point and infrastructure variation prevent exact repeatability.
The consequence for engineering is that a single observation proves very little. One successful run does not establish that a behaviour is reliable; one failure does not establish that it is broken. Anything worth claiming needs repetition, and the number of repetitions needed scales with how rare the failure is — a fault occurring one run in ten needs roughly twenty clean runs before "fixed" means anything. This is exactly the arithmetic that makes a Flaky Test so expensive.
It also changes what an assertion can say. An evaluation must assert what the system guarantees rather than what a good run produces: bounds that hold across rolls, invariants the code enforces, counts the pipeline cannot violate. Numeric expectations tuned to one recording break on the next, and the breakage reads as a regression.
The standard mitigations are to make the deterministic parts genuinely deterministic — Record and Replay Testing for the model call, fixed seeds for anything random, pinned versions — so that variance is isolated to the one place it is unavoidable. See Determinism.
See also5
Hand-picked in the note itself — the neighbours worth reading next.
Evaluation Harness
A repeatable test suite for model behaviour, since prompts and tool descriptions cannot be type-checked.
Agents & Language Models11 connections
Large Language Model
A neural network trained to predict text, used as a general-purpose instruction-following system.
Agents & Language Models22 connections
Flaky Test
A test that passes and fails on identical input, and the most corrosive thing a suite can contain.
Testing & Verification12 connections
Assertion
The statement in a test that must hold, and the only part that can actually fail.
Testing & Verification9 connections
LLM-as-Judge
Using a language model to score another model's output against a rubric.
Agents & Language Models10 connections
Related2
Nearby in the graph rather than deliberately chosen. Looser, sometimes surprising.
Linked from16
Notes elsewhere in the wiki that reach for this one.
- Anchoring EffectMethod
A stated number pulls subsequent estimates toward it, regardless of relevance.
- AssertionTesting & Verification
The statement in a test that must hold, and the only part that can actually fail.
- DeterminismSystems & Tooling
The property that identical inputs produce identical outputs, and the foundation of caching and verification.
- Evaluation HarnessAgents & Language Models
A repeatable test suite for model behaviour, since prompts and tool descriptions cannot be type-checked.
- FingerprintAgents & Language Models
A hash over everything that determined a result, used to detect when the result has gone stale.
- Flaky TestTesting & Verification
A test that passes and fails on identical input, and the most corrosive thing a suite can contain.
- HallucinationAgents & Language Models
Fluent, confident output that is not grounded in anything real.
- Large Language ModelAgents & Language Models
A neural network trained to predict text, used as a general-purpose instruction-following system.
- LLM-as-JudgeAgents & Language Models
Using a language model to score another model's output against a rubric.
- Model RoutingAgents & Language Models
Choosing which model handles which step, trading capability against cost and latency.
- Prompt EngineeringAgents & Language Models
Shaping a model's input to make the desired behaviour reliable rather than occasional.
- Record and Replay TestingAgents & Language Models
Capturing a real interaction once and replaying it deterministically in later test runs.
- Structured OutputAgents & Language Models
Constraining a model to emit data conforming to a declared schema rather than free text.
- Test DoubleTesting & Verification
A stand-in for a real dependency during a test — and the ways a stand-in can lie.
- Test FixtureTesting & Verification
The prepared data or state a test runs against.
- TokenAgents & Language Models
The sub-word unit a language model actually reads and writes; the unit of cost and of context.