Statistics, experimentation, causal reasoning · insight-0031

Expect the worst performers to improve without treatment and keep an untreated comparison

Situation
Operations selects the ten routes with the worst on-time performance last quarter for a new turnaround process, and reports their improvement next quarter.
Action
Randomize the process among the eligible poor performers, or compare with similar poor performers that were not treated. If neither is possible, select on a multi-period average and report the expected regression from historical data.
Reason
Extreme observations combine true level and luck; the luck does not repeat, so remeasurement drifts toward the mean regardless of action.
Boundary
Interventions on worst stores, riskiest customers, slowest agents and highest-error forecasts.

Symptom or risk

Units chosen for extreme values tend to move toward the average on remeasurement, so the process gets credit for regression to the mean.

Persistent, stable metrics with little noise show little regression to the mean.

Holding back some poor performers delays help for them; the comparison makes the result credible.

Comparing the selected group's after-value with its own before-value only.

Regression to the mean can make natural variation look like real change; it is stronger with more measurement error and when follow-up is restricted to a subgroup selected for extreme baseline values.
Regression to the mean: what it is and how to deal with it