Selection Bias
A sample distorted because inclusion in it was never random.
Selection bias is what happens when the process that put items into a sample is correlated with the thing you are trying to measure, so the sample no longer represents the population it is supposed to stand in for. Survivorship Bias is one shape of it — inclusion determined by having survived — but the family is broader.
Self-selection distorts a survey when the people who bother to respond differ systematically from the people who do not: users who file a bug report are, by definition, the ones patient enough to file one. Attrition distorts a longitudinal study when the participants who drop out share a trait related to the outcome, so the group that remains looks healthier, happier, or more successful than the group that started. Collider bias is the subtlest form: conditioning analysis on a variable that is itself a common effect of two other variables can manufacture a correlation between causes that have no real relationship, purely from the act of selecting on their shared consequence.
The counterintuitive consequence is that a larger biased sample is not safer than a smaller one — it is worse, because it produces a narrower confidence interval around the wrong number, which reads as more certainty rather than less. Checking for selection effects means asking how a record ends up observed at all, not just what the observed records say, in the same spirit as Ground Truth and the diagnostic habit behind Sensitivity and Specificity.
See also8
Percentage Point
The unit of an absolute change in a percentage, and the gap between that and a relative change.
Method10 connections
Streetlight Effect
Looking where the light is good rather than where the answer is.
Method12 connections
Fermi Estimation
Reaching a defensible order-of-magnitude answer by decomposing a question into estimable factors.
Method24 connections
Truncation Bias
Reading a truncated result as if it were the whole result, so every counterexample is invisible.
Method28 connections
Automation Bias
Over-trusting an automated recommendation, including against available contrary evidence.
Method13 connections
Online Review
Ratings and written opinions that customers post about products, places and services, which have become one of the main inputs to buying decisions and, for the same reason, a constant target for fakes, bribes and coordinated attacks.
Web Platform7 connections
Statistical Sampling
Learning about a whole population by measuring a randomly chosen part of it, where the way the sample is drawn matters far more than how large it is.
Method5 connections
Web Analytics
Measuring how people use a website, from parsing server logs to JavaScript page tags and Google's free Analytics, with metrics like pageviews and bounce rate that are distorted by bots, blockers, and consent rules and easy to optimize in the wrong direction.
Web Platform11 connections
Linked from14
- Availability HeuristicMethod
Judging how likely or common something is by how easily examples come to mind, not by actual frequency.
- Base Rate FallacyMethod
judging a specific case from vivid evidence while discounting how common the categories actually are.
- Betteridge's Law of HeadlinesMethod
any headline phrased as a question can safely be answered "no".
- Confidence IntervalMethod
A range around an estimate that conveys how precisely it was measured, more informative than a yes-or-no significance verdict, and easy to misread as a statement of probability about the true value.
- Confirmation BiasMethod
The tendency to search for, interpret, and recall evidence in ways that favor what you already believe.
- Online ReviewWeb Platform
Ratings and written opinions that customers post about products, places and services, which have become one of the main inputs to buying decisions and, for the same reason, a constant target for fakes, bribes and coordinated attacks.
- P-HackingMeaning & Society
Trying enough analyses on the same data until one crosses the significance threshold, then reporting only that one.
- Simpson's ParadoxMethod
A trend appears in several groups of data but reverses or disappears when the groups are combined.
- Statistical SamplingMethod
Learning about a whole population by measuring a randomly chosen part of it, where the way the sample is drawn matters far more than how large it is.
- Streetlight EffectMethod
Looking where the light is good rather than where the answer is.
- Survivorship BiasMethod
Drawing conclusions from a sample already filtered by success.
- Tier ListPlay & Games
A community-authored ordinal ranking of a game's options, and what the ranking actually measures.
- Universal Basic IncomeMeaning & Society
An unconditional cash payment to every member of a population, and its contested design and evidence.
- Web AnalyticsWeb Platform
Measuring how people use a website, from parsing server logs to JavaScript page tags and Google's free Analytics, with metrics like pageviews and bounce rate that are distorted by bots, blockers, and consent rules and easy to optimize in the wrong direction.