Observability
How much of a system's internal state can be inferred from what it emits.
Observability is the degree to which a system's internal state can be determined from its outputs — logs, metrics, traces, and whatever else it emits. The term comes from control theory; in software it names the difference between a system you can ask questions of and one you can only guess about.
It is distinct from monitoring. Monitoring answers questions you thought to ask in advance ("is the error rate above two percent?"). Observability is what lets you answer a question nobody anticipated, which is what real incidents demand.
Practically it is built from small deliberate choices. Emitting the resolved configuration at startup rather than assuming it. Logging which credential a tool actually used rather than which one you configured. Recording free disk space at the moment a job fails, so the next occurrence is diagnosable rather than mysterious. Writing a decision log for a gate so "it skipped" and "it passed" are distinguishable afterwards.
Poor observability produces a characteristic waste: hours spent reconstructing state that the system could have told you in one line. It is also what makes Silent Failure possible — a failure with no emission is invisible by construction.
See also8
Hand-picked in the note itself — the neighbours worth reading next.
Instrumentation
Adding measurement to code so its execution can be observed, and the distortions that introduces.
Testing & Verification12 connections
Ground Truth
The measurement that arbitrates, as opposed to the artifact that merely reports.
Method30 connections
Silent Failure
A failure that produces no signal, so the absence of an error reads as success.
Method44 connections
Reproducible Case
The smallest set of steps that reliably produces a fault, and the unit of real debugging.
Method17 connections
Resource Starvation
Failures caused by competition for a finite shared resource rather than by any defect.
Testing & Verification10 connections
Root Cause Analysis
Escalating past the visible symptom until you find the layer that actually produced it.
Method21 connections
Cron
Time-based scheduling of recurring jobs, and the reasons scheduled jobs quietly stop working.
Systems & Tooling11 connections
Fail Fast
Crash at the point of misconfiguration rather than degrading quietly into a wrong state.
Method12 connections
Related2
Nearby in the graph rather than deliberately chosen. Looser, sometimes surprising.
Linked from14
Notes elsewhere in the wiki that reach for this one.
- Code SigningSystems & Tooling
Cryptographically attesting who produced a binary, and the operating-system checks built on it.
- CronSystems & Tooling
Time-based scheduling of recurring jobs, and the reasons scheduled jobs quietly stop working.
- DaemonSystems & Tooling
A long-running background process managed by the operating system's service supervisor.
- Fail FastMethod
Crash at the point of misconfiguration rather than degrading quietly into a wrong state.
- Ground TruthMethod
The measurement that arbitrates, as opposed to the artifact that merely reports.
- InstrumentationTesting & Verification
Adding measurement to code so its execution can be observed, and the distortions that introduces.
- ProcessSystems & Tooling
A running program with its own memory, environment, and identity.
- Reproducible CaseMethod
The smallest set of steps that reliably produces a fault, and the unit of real debugging.
- Resource StarvationTesting & Verification
Failures caused by competition for a finite shared resource rather than by any defect.
- Root Cause AnalysisMethod
Escalating past the visible symptom until you find the layer that actually produced it.
- Secret ManagementSystems & Tooling
Storing, distributing, and rotating credentials without embedding them in code or history.
- Silent FailureMethod
A failure that produces no signal, so the absence of an error reads as success.
- Streaming ResponseNetworks & Distribution
Delivering a response incrementally as it is produced rather than all at once when complete.
- Truncation BiasMethod
Reading a truncated result as if it were the whole result, so every counterexample is invisible.