Statistics, experimentation, causal reasoning · insight-1014
Estimate many route or segment effects with partial pooling instead of testing each one separately
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist
- Industries
- aviation, retail, public-sector
Summary: situation, action, reason, boundary
Situation
A test is read out for 40 origin markets with a few thousand passengers each, and the analyst lists those with a raw p below 0.05, or those that survive a correction.
Action
Fit a multilevel model with market-level random effects, or apply empirical-Bayes shrinkage, and report shrunken estimates with intervals plus the between-market spread. Read the overall effect and the spread first, then individual markets.
Reason
Partial pooling moves noisy estimates toward the group mean, which improves efficiency and reduces the exaggeration of extreme markets without widening every interval.
Boundary
Many comparable groups with moderate data each.
Symptom or risk
Raw market estimates are noisy and the extremes are selected, while strict corrections wipe everything out; the table is both overconfident and useless.
Exceptions
Fewer than about five groups, or groups that are different by design and should not borrow strength.
Trade-offs
Shrinkage biases the estimate of a genuinely exceptional market toward the average.
Pitfalls
Treating shrunken values as confirmed effects of individual markets.
Lessons and concepts
Claims and their sources
Multilevel models that partially pool estimates toward one another address the multiple-comparisons concern and produce more efficient estimates, whereas classical procedures keep interval centres fixed and widen the intervals.