Flaky Test
A test that passes and fails on identical input, and the most corrosive thing a suite can contain.
A flaky test fails intermittently on unchanged code. Its real cost is not the failed run but the habit it teaches: once a suite has flakes, every failure gets a rerun before it gets a diagnosis, and real defects ship because they looked like flakes.
Flakes have a small number of causes, and identifying which one is at play is most of the work.
Timing. An assertion racing an asynchronous update. Waiting for a condition rather than a duration fixes it; a fixed sleep only moves it. Ordering. Shared state between tests, so a test passes alone and fails after a sibling. Contention. A timeout that only blows under load, which is Resource Starvation and not a defect at all. Environment. Disk exhaustion, a cold build cache, a development server holding a port. Non-determinism in the system under test — clock, randomness, iteration order.
Two diagnostic rules earn their keep. An identical failure across repeated runs of the same revision is a real gap, not a flake; rerunning cannot make it pass. A failure that disappears the moment logging or a debugger is added is a Heisenbug, and the instinct to instrument harder is what keeps it hidden. And a flake at a ten percent rate needs roughly twenty consecutive green runs before "fixed" means anything — one green run proves nothing. See Nondeterminism.
See also12
Visual Regression Testing
Comparing rendered output against a stored baseline to catch unintended visual change.
Testing & Verification14 connections
Race Condition
A defect whose occurrence depends on the relative timing of concurrent operations.
Testing & Verification36 connections
Determinism
The property that identical inputs produce identical outputs, and the foundation of caching and verification.
Systems & Tooling61 connections
Reproducible Case
The smallest set of steps that reliably produces a fault, and the unit of real debugging.
Method27 connections
Bundler
The tool that resolves an application's modules into the files a browser downloads.
Web Platform18 connections
Instrumentation
Adding measurement to code so its execution can be observed, and the distortions that introduces.
Testing & Verification19 connections
Regression
A previously working behavior that a change has broken.
Method23 connections
Ground Truth
The measurement that arbitrates, as opposed to the artifact that merely reports.
Method99 connections
Fake Clock
Replacing the real clock in tests with one the test controls, so time-dependent code can be checked at midnight, across DST, or after an hour of retries without waiting.
Testing & Verification9 connections
Developer Experience
How it feels to build software with a given set of tools, codebase, and process, shaped mostly by how fast feedback arrives, how much a developer must hold in mind, and how often they can work uninterrupted.
Version Control & Delivery7 connections
Test Isolation
The property that each test passes or fails on its own, regardless of which other tests run, in what order, or in parallel.
Testing & Verification20 connections
Service Virtualization
Replacing a real external service with a simulated one that speaks the same protocol over the network, so tests can run without the real dependency and can reproduce its failures on demand.
Testing & Verification15 connections
Linked from28
- Branch ProtectionVersion Control & Delivery
Server-side rules that stop people from pushing straight to important branches, requiring reviews, passing checks, and intact history instead.
- BundlerWeb Platform
The tool that resolves an application's modules into the files a browser downloads.
- Continuous IntegrationVersion Control & Delivery
Automatically building and testing every change, so integration problems surface immediately.
- DeterminismSystems & Tooling
The property that identical inputs produce identical outputs, and the foundation of caching and verification.
- Developer ExperienceVersion Control & Delivery
How it feels to build software with a given set of tools, codebase, and process, shaped mostly by how fast feedback arrives, how much a developer must hold in mind, and how often they can work uninterrupted.
- End-to-End TestTesting & Verification
A test that drives the whole running system the way a user would, from the interface down through every real dependency, buying the most confidence at the highest cost.
- Exponential BackoffNetworks & Distribution
Waiting longer after each failed attempt, with a dose of randomness, so retries give a struggling system room to recover instead of hammering it.
- Fake ClockTesting & Verification
Replacing the real clock in tests with one the test controls, so time-dependent code can be checked at midnight, across DST, or after an hour of retries without waiting.
- Git BisectTesting & Verification
Finding the commit that introduced a bug by binary search through history, testing the midpoint between a known-good and a known-bad version until one commit remains.
- HeisenbugTesting & Verification
A fault that changes or disappears when someone tries to observe it.
- InstrumentationTesting & Verification
Adding measurement to code so its execution can be observed, and the distortions that introduces.
- Least Recently Used CacheComputation & Algorithms
A fixed-capacity cache that evicts whichever entry has gone untouched the longest.
- Minimal Reproducible ExampleTesting & Verification
The smallest self-contained program or set of steps that still shows a bug, and why making one is often the moment the bug gets understood.
- NondeterminismAgents & Language Models
The property that identical inputs may produce different outputs, and what it costs to test around.
- Pure FunctionComputation & Algorithms
A function whose output depends only on its input and which causes no observable effect outside itself.
- Race ConditionTesting & Verification
A defect whose occurrence depends on the relative timing of concurrent operations.
- Record and ReplayTesting & Verification
Capturing real network responses once and replaying them in later test runs, which makes tests fast and deterministic but lets the recordings drift from the real service.
- RegressionMethod
A previously working behavior that a change has broken.
- Regression TestingTesting & Verification
Rerunning existing tests after every change to confirm that what used to work still works, how the suite grows one fixed bug at a time, and how teams keep it fast enough to run.
- Reproducible CaseMethod
The smallest set of steps that reliably produces a fault, and the unit of real debugging.
- Resource StarvationTesting & Verification
Failures caused by competition for a finite shared resource rather than by any defect.
- Service VirtualizationTesting & Verification
Replacing a real external service with a simulated one that speaks the same protocol over the network, so tests can run without the real dependency and can reproduce its failures on demand.
- Smoke TestTesting & Verification
A fast, shallow check that a build isn't catastrophically broken, run before investing in a deeper test pass.
- Test Data BuilderTesting & Verification
Helpers that build valid test objects from sensible defaults so each test states only the fields it cares about, and the Object Mother pattern they grew out of.
- Test Impact AnalysisTesting & Verification
Running only the tests a change could plausibly affect, using dependency graphs, coverage maps, or learned models, so large suites stay fast without giving up their safety net.
- Test IsolationTesting & Verification
The property that each test passes or fails on its own, regardless of which other tests run, in what order, or in parallel.
- Time ZoneSystems & Tooling
Civil time is a political layer over solar time, and its rules change often enough to be data.
- Vacuous TruthMethod
A statement that holds only because its subject set is empty — a green result that measured nothing.