Statistics, experimentation, causal reasoning · insight-1029
Report the effective sample size after weighting, because large weight variation shrinks it
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data analyst, Data scientist
- Industries
- aviation, public-sector, retail
Summary: situation, action, reason, boundary
Situation
A satisfaction survey is weighted to the passenger mix by route and cabin, and the report keeps quoting the raw respondent count in its margin of error.
Action
Compute the effective sample size as the squared sum of weights divided by the sum of squared weights, inspect the largest weights, cap or collapse sparse cells, and base margins of error on the effective size. Include in the weights the variables that drive inclusion and non-response.
Reason
Unequal weights act like discarding data; the more they vary, the fewer observations the estimate is worth.
Boundary
Post-stratified or raked surveys and weighted panels.
Symptom or risk
A few heavily weighted respondents dominate, so the real precision is lower than the headline count suggests and small differences are over-read.
Exceptions
Near-equal weights, where the loss is negligible.
Trade-offs
Capping weights reduces variance but leaves some bias.
Pitfalls
Weighting on variables for which the population totals are themselves uncertain.
Example
Hypothetical: 900 respondents at weight 0.8 and 100 at 2.8 (total weight 1,000) give an effective size of about 735, not 1,000.
Lessons and concepts
Claims and their sources
Models of survey responses should account for the variables that drive survey inclusion, nonresponse, weighting and clustering, which can require very many poststratification cells.