Statistical selection effect
Regression to the Mean
When two measurements are imperfectly correlated, units selected for an extreme first value tend to be less extreme on a subsequent measurement.
E[Z2 | Z1 = z] = r z when standardized in a linear model
The shrinkage factor is the correlation r, not a force pulling observations toward average. Real change, measurement error, and selection can coexist.
Extreme first scores send translucent paths toward expected retests; stronger reliability preserves more of the original extremity.
()
The animation runs automatically, pauses on the conclusion, and then repeats. The main control changes the scenario rather than scrubbing the timeline.
- CHANGE
- Test-retest reliability
- WATCH
- expected retest
- MEANING
- Extreme first scores send translucent paths toward expected retests; stronger reliability preserves more of the original extremity.
Select on an extreme, then watch imperfect correlation reveal itself.
A first-test versus retest scatterplot highlights the selected tail and its conditional average below the identity line.
What it actually says
Regression to the mean follows from conditional selection when repeated outcomes contain both persistent and nonpersistent components. It needs no compensating mechanism.
It is a central threat in before-and-after studies: patients, schools, athletes, or firms are often chosen precisely because the baseline was unusually bad or good.
"A useful law compresses a pattern. It does not erase the conditions that make the pattern true."
How the idea developed
The modern form emerged through observation, argument, and later refinement. The timeline separates the first insight from the version now used in textbooks and practice.[1]
Francis Galton reports regression in hereditary stature.
The concept becomes foundational in correlation and experimental design.
Trials and causal studies use controls to separate treatment effects from regression.
How the pattern works
The relation becomes useful only when its mechanism, measurement process, and operating range are visible.
Cases enter because the first measure is unusual.
Transient components do not repeat fully.
Expected follow-up is closer to the population mean.
The shrinkage factor is the correlation r, not a force pulling observations toward average. Real change, measurement error, and selection can coexist.
Where it earns its keep
Applications are strongest when the law changes a decision, measurement, model, or experiment rather than merely providing an analogy.
Use a comparison group
ApplicationControls experience the same selection and time process.
Avoid naive before-after attribution.
Do not punish or praise noise
ApplicationExtreme performance often moderates naturally.
Estimate reliability.
Where it stops working
Population drift, treatment, learning, seasonality, and heterogeneous subgroups can also change follow-up values and require separate modeling.
"The mean causes the change"
Better: Regression is a conditional pattern, not a restoring force."Every extreme must reverse"
Better: Individual outcomes can become more extreme; the statement concerns expectation.Sources and further reading
Original publications and serious secondary scholarship are prioritized over summaries.
- Galton - Regression Towards Mediocrity in Hereditary StaturePrimary 1886 paper.https://galton.org/essays/1880-1889/galton-1886-jaigi-regression-stature.pdf
- Barnett, van der Pols, and Dobson - Regression to the MeanMethodological review.https://doi.org/10.1093/ije/dyh299
- BMJ - Statistics Notes: Regression Towards the MeanClinical research explanation.https://www.bmj.com/content/308/6942/1499