Evaluation Harness
A repeatable test suite for model behavior, since prompts and tool descriptions cannot be type-checked.
An evaluation harness is a test suite for a system built on a Large Language Model. It exists because the parts of such a system that most affect behavior — the System Prompt, tool descriptions, schema field text — are prose, and no compiler, linter, or unit test can tell you whether an edit to them helped.
An eval defines scenarios (an input situation and an intent), runs the system, and asserts on what happened. The most valuable assertions are usually about routing — which tool was called, whether anything was called at all — because that is where prompt changes show up. A behavior change frequently appears as an empty list of tool calls rather than as anything visibly wrong in the reply text.
Designing a scenario well is harder than it looks. An ask that brushes a policy rule forks: one run complies, the next declines, and both are correct — so the recording measures policy, not routing. A stubbed backend that lets a create succeed while the corresponding read returns nothing teaches the model its write failed, and it writes again; now the recording measures the stub. Before recording, ask what else the scenario could legitimately be measuring.
Live evaluation costs money and time, which is why most harnesses replay; see Record and Replay Testing.
See also9
LLM-as-Judge
Using a language model to score another model's output against a rubric.
Agents & Language Models12 connections
Fingerprint
A hash over everything that determined a result, used to detect when the result has gone stale.
Agents & Language Models12 connections
Nondeterminism
The property that identical inputs may produce different outputs, and what it costs to test around.
Agents & Language Models24 connections
Prompt Engineering
Shaping a model's input to make the desired behavior reliable rather than occasional.
Agents & Language Models22 connections
Guardrail
A rule constraining what a model may do, enforced by prompt, schema, or surrounding code.
Agents & Language Models17 connections
Tool Use
Giving a model a set of callable functions so it can act on the world rather than only describe it.
Agents & Language Models27 connections
Test Double
A stand-in for a real dependency during a test — and the ways a stand-in can lie.
Testing & Verification18 connections
Model Card
A short standard document published alongside a machine learning model describing what it's for, how it was trained and evaluated, where it performs worse, and what it shouldn't be used for, now routine on model hubs and for frontier releases.
Agents & Language Models5 connections
Ablation Study
An experiment that removes or disables one part of a system and measures what changes, borrowed from neuroscience lesion studies and now the standard way machine learning papers show which of their components actually matter.
Agents & Language Models3 connections
Linked from17
- Ablation StudyAgents & Language Models
An experiment that removes or disables one part of a system and measures what changes, borrowed from neuroscience lesion studies and now the standard way machine learning papers show which of their components actually matter.
- Automatic Speech RecognitionAgents & Language Models
Turning audio into text, and the reasons real-world accuracy is set by the signal more than by the model.
- FingerprintAgents & Language Models
A hash over everything that determined a result, used to detect when the result has gone stale.
- GuardrailAgents & Language Models
A rule constraining what a model may do, enforced by prompt, schema, or surrounding code.
- Large Language ModelAgents & Language Models
A neural network trained to predict text, used as a general-purpose instruction-following system.
- LLM-as-JudgeAgents & Language Models
Using a language model to score another model's output against a rubric.
- Mechanistic InterpretabilityAgents & Language Models
Reverse-engineering the computation a trained network learned into human-legible features and circuits.
- Model CardAgents & Language Models
A short standard document published alongside a machine learning model describing what it's for, how it was trained and evaluated, where it performs worse, and what it shouldn't be used for, now routine on model hubs and for frontier releases.
- Model RoutingAgents & Language Models
Choosing which model handles which step, trading capability against cost and latency.
- NondeterminismAgents & Language Models
The property that identical inputs may produce different outputs, and what it costs to test around.
- Precision and RecallAgents & Language Models
The two measures of how well a classifier or search finds what it should, how often its hits are right and how many of the real cases it catches, and why improving one usually costs the other.
- Prompt EngineeringAgents & Language Models
Shaping a model's input to make the desired behavior reliable rather than occasional.
- Record and Replay TestingAgents & Language Models
Capturing a real interaction once and replaying it deterministically in later test runs.
- RerankingAgents & Language Models
Retrieving many candidates cheaply and then reordering a short list with a slower, more accurate model, the standard way search and RAG systems get relevant results into a limited context.
- System PromptAgents & Language Models
The standing instructions that frame every turn of a conversation with a model.
- Test DoubleTesting & Verification
A stand-in for a real dependency during a test — and the ways a stand-in can lie.
- Test OracleTesting & Verification
Whatever decides whether a test's output is correct, and the often-overlooked hard part of testing when nobody knows the right answer in advance.