Evaluation Harness
A repeatable test suite for model behavior, since prompts and tool descriptions cannot be type-checked.
An evaluation harness is a test suite for a system built on a Large Language Model. It exists because the parts of such a system that most affect behavior — the System Prompt, tool descriptions, schema field text — are prose, and no compiler, linter, or unit test can tell you whether an edit to them helped.
An eval defines scenarios (an input situation and an intent), runs the system, and asserts on what happened. The most valuable assertions are usually about routing — which tool was called, whether anything was called at all — because that is where prompt changes show up. A behavior change frequently appears as an empty list of tool calls rather than as anything visibly wrong in the reply text.
Designing a scenario well is harder than it looks. An ask that brushes a policy rule forks: one run complies, the next declines, and both are correct — so the recording measures policy, not routing. A stubbed backend that lets a create succeed while the corresponding read returns nothing teaches the model its write failed, and it writes again; now the recording measures the stub. Before recording, ask what else the scenario could legitimately be measuring.
Live evaluation costs money and time, which is why most harnesses replay; see Record and Replay Testing.
See also5
LLM-as-Judge
Using a language model to score another model's output against a rubric.
Agents & Language Models10 connections
Fingerprint
A hash over everything that determined a result, used to detect when the result has gone stale.
Agents & Language Models12 connections
Nondeterminism
The property that identical inputs may produce different outputs, and what it costs to test around.
Agents & Language Models23 connections
Prompt Engineering
Shaping a model's input to make the desired behavior reliable rather than occasional.
Agents & Language Models21 connections
Guardrail
A rule constraining what a model may do, enforced by prompt, schema, or surrounding code.
Agents & Language Models14 connections
Related2
Nearby in the graph rather than deliberately chosen. Looser, sometimes surprising.
Linked from12
- Automatic Speech RecognitionAgents & Language Models
Turning audio into text, and the reasons real-world accuracy is set by the signal more than by the model.
- FingerprintAgents & Language Models
A hash over everything that determined a result, used to detect when the result has gone stale.
- GuardrailAgents & Language Models
A rule constraining what a model may do, enforced by prompt, schema, or surrounding code.
- Large Language ModelAgents & Language Models
A neural network trained to predict text, used as a general-purpose instruction-following system.
- LLM-as-JudgeAgents & Language Models
Using a language model to score another model's output against a rubric.
- Mechanistic InterpretabilityAgents & Language Models
Reverse-engineering the computation a trained network learned into human-legible features and circuits.
- Model RoutingAgents & Language Models
Choosing which model handles which step, trading capability against cost and latency.
- NondeterminismAgents & Language Models
The property that identical inputs may produce different outputs, and what it costs to test around.
- Prompt EngineeringAgents & Language Models
Shaping a model's input to make the desired behavior reliable rather than occasional.
- Record and Replay TestingAgents & Language Models
Capturing a real interaction once and replaying it deterministically in later test runs.
- System PromptAgents & Language Models
The standing instructions that frame every turn of a conversation with a model.
- Test DoubleTesting & Verification
A stand-in for a real dependency during a test — and the ways a stand-in can lie.