Statistics, experimentation, causal reasoning · insight-1023
Inspect propensity-score overlap and weight extremes before reporting an inverse-probability-weighted effect
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- healthcare, banking, retail
Summary: situation, action, reason, boundary
Situation
An observational analysis of upgrade offers uses inverse-probability weights, and a handful of customers carry weights above 50.
Action
Plot propensity distributions by arm, tabulate weight quantiles and the maximum, and check weighted covariate balance with standardised differences. Trim or restrict to the overlap region, and say that the estimand changed.
Reason
Weighting makes treatment independent of the measured covariates only where every covariate pattern can receive either arm, and balance after weighting is the check that it worked.
Boundary
Observational comparisons using propensity scores, marginal structural models or weighted regression.
Symptom or risk
Some customer types almost never or always receive the offer, so the estimate rests on a few observations and its variance is understated or erratic.
Exceptions
Randomised tests, where propensities are known.
Trade-offs
Trimming trades bias for variance and narrows who the answer is about.
Pitfalls
Checking balance before weighting and assuming weighting fixed it.
Lessons and concepts
Claims and their sources
Inverse probability of treatment weighting creates a pseudo-sample in which treatment assignment no longer depends on measured covariates, and the authors propose qualitative and quantitative balance diagnostics for the weighted sample.