Kelly Mears

Flaky Test

A test that passes and fails on identical input, and the most corrosive thing a suite can contain.

Testing & Verification2 min read224 words15 out · 28 in
also calledFlakeIntermittent failure

A flaky test fails intermittently on unchanged code. Its real cost is not the failed run but the habit it teaches: once a suite has flakes, every failure gets a rerun before it gets a diagnosis, and real defects ship because they looked like flakes.

Flakes have a small number of causes, and identifying which one is at play is most of the work.

Timing. An assertion racing an asynchronous update. Waiting for a condition rather than a duration fixes it; a fixed sleep only moves it. Ordering. Shared state between tests, so a test passes alone and fails after a sibling. Contention. A timeout that only blows under load, which is Resource Starvation and not a defect at all. Environment. Disk exhaustion, a cold build cache, a development server holding a port. Non-determinism in the system under test — clock, randomness, iteration order.

Two diagnostic rules earn their keep. An identical failure across repeated runs of the same revision is a real gap, not a flake; rerunning cannot make it pass. A failure that disappears the moment logging or a debugger is added is a Heisenbug, and the instinct to instrument harder is what keeps it hidden. And a flake at a ten percent rate needs roughly twenty consecutive green runs before "fixed" means anything — one green run proves nothing. See Nondeterminism.

See also12

Visual Regression Testing

Comparing rendered output against a stored baseline to catch unintended visual change.

Testing & Verification14 connections

Race Condition

A defect whose occurrence depends on the relative timing of concurrent operations.

Testing & Verification36 connections

Determinism

The property that identical inputs produce identical outputs, and the foundation of caching and verification.

Systems & Tooling61 connections

Reproducible Case

The smallest set of steps that reliably produces a fault, and the unit of real debugging.

Method27 connections

Bundler

The tool that resolves an application's modules into the files a browser downloads.

Web Platform18 connections

Instrumentation

Adding measurement to code so its execution can be observed, and the distortions that introduces.

Testing & Verification19 connections

Regression

A previously working behavior that a change has broken.

Method23 connections

Ground Truth

The measurement that arbitrates, as opposed to the artifact that merely reports.

Method99 connections

Fake Clock

Replacing the real clock in tests with one the test controls, so time-dependent code can be checked at midnight, across DST, or after an hour of retries without waiting.

Testing & Verification9 connections

Developer Experience

How it feels to build software with a given set of tools, codebase, and process, shaped mostly by how fast feedback arrives, how much a developer must hold in mind, and how often they can work uninterrupted.

Version Control & Delivery7 connections

Test Isolation

The property that each test passes or fails on its own, regardless of which other tests run, in what order, or in parallel.

Testing & Verification20 connections

Service Virtualization

Replacing a real external service with a simulated one that speaks the same protocol over the network, so tests can run without the real dependency and can reproduce its failures on demand.

Testing & Verification15 connections

Linked from28