Evaluation Harness
A repeatable test suite for model behaviour, since prompts and tool descriptions cannot be type-checked.
An evaluation harness is a test suite for a system built on a Large Language Model. It exists because the parts of such a system that most affect behaviour — the System Prompt, tool descriptions, schema field text — are prose, and no compiler, linter, or unit test can tell you whether an edit to them helped.
An eval defines scenarios (an input situation and an intent), runs the system, and asserts on what happened. The most valuable assertions are usually about routing — which tool was called, whether anything was called at all — because that is where prompt changes show up. A behaviour change frequently appears as an empty list of tool calls rather than as anything visibly wrong in the reply text.
Designing a scenario well is harder than it looks. An ask that brushes a policy rule forks: one run complies, the next declines, and both are correct — so the recording measures policy, not routing. A stubbed backend that lets a create succeed while the corresponding read returns nothing teaches the model its write failed, and it writes again; now the recording measures the stub. Before recording, ask what else the scenario could legitimately be measuring.
Live evaluation costs money and time, which is why most harnesses replay; see Record and Replay Testing.
See also5
Hand-picked in the note itself — the neighbours worth reading next.
LLM-as-Judge
Using a language model to score another model's output against a rubric.
Agents & Language Models10 connections
Fingerprint
A hash over everything that determined a result, used to detect when the result has gone stale.
Agents & Language Models11 connections
Nondeterminism
The property that identical inputs may produce different outputs, and what it costs to test around.
Agents & Language Models17 connections
Prompt Engineering
Shaping a model's input to make the desired behaviour reliable rather than occasional.
Agents & Language Models18 connections
Guardrail
A rule constraining what a model may do, enforced by prompt, schema, or surrounding code.
Agents & Language Models13 connections
Related2
Nearby in the graph rather than deliberately chosen. Looser, sometimes surprising.
Linked from10
Notes elsewhere in the wiki that reach for this one.
- FingerprintAgents & Language Models
A hash over everything that determined a result, used to detect when the result has gone stale.
- GuardrailAgents & Language Models
A rule constraining what a model may do, enforced by prompt, schema, or surrounding code.
- Large Language ModelAgents & Language Models
A neural network trained to predict text, used as a general-purpose instruction-following system.
- LLM-as-JudgeAgents & Language Models
Using a language model to score another model's output against a rubric.
- Model RoutingAgents & Language Models
Choosing which model handles which step, trading capability against cost and latency.
- NondeterminismAgents & Language Models
The property that identical inputs may produce different outputs, and what it costs to test around.
- Prompt EngineeringAgents & Language Models
Shaping a model's input to make the desired behaviour reliable rather than occasional.
- Record and Replay TestingAgents & Language Models
Capturing a real interaction once and replaying it deterministically in later test runs.
- System PromptAgents & Language Models
The standing instructions that frame every turn of a conversation with a model.
- Test DoubleTesting & Verification
A stand-in for a real dependency during a test — and the ways a stand-in can lie.