Statistics, experimentation, causal reasoning · insight-0035
Do not rank small groups by raw rates; show intervals or shrink toward the overall rate
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data analyst, Data scientist
- Industries
- aviation, retail, healthcare, cross-industry
Summary: situation, action, reason, boundary
Situation
A league table ranks sales agents, routes or stores by conversion rate, and the top and bottom places go to units with very few observations.
Action
Show each rate with a confidence interval or plot it against volume with control limits (a funnel plot). For ranking, use empirical-Bayes or hierarchical estimates that pull small groups toward the overall rate, and set a minimum volume for inclusion.
Reason
The sampling variance of a proportion falls with sample size; extremes concentrate in the smallest groups even when all have the same true rate.
Boundary
Agent performance, route conversion, store returns, A/B segment tables and supplier quality.
Symptom or risk
Small denominators produce extreme rates by chance, so rewards and interventions target noise.
Exceptions
Groups with large and similar volumes can be compared on raw rates.
Trade-offs
Shrinkage biases truly exceptional small groups toward average; review them separately.
Pitfalls
Excluding small groups silently, which hides new units and survivorship.