Statistics, experimentation, causal reasoning · insight-1017
Test for sorting around the cutoff before trusting a regression discontinuity estimate
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist
- Industries
- aviation, banking, public-sector
Summary: situation, action, reason, boundary
Situation
A status tier starts at a points threshold (hypothetical: 25,000), and members just above are compared with those just below to estimate the effect of lounge access.
Action
Plot the running variable near the cutoff, run a density-continuity test, and check pre-treatment covariates for jumps at the threshold. If there is heaping or a spike, exclude a donut around the cutoff or abandon the design.
Reason
The design needs potential outcomes to be continuous at the threshold, and that fails when units can manipulate their score.
Boundary
Loyalty tiers, eligibility thresholds, score-based approvals.
Symptom or risk
Customers who can steer their points, for example by topping up before the deadline, sort themselves over the line, so the two groups differ in motivation.
Exceptions
Thresholds applied automatically on data customers cannot influence, for example age at booking.
Trade-offs
A donut hole removes the nearest, most comparable observations and widens intervals.
Pitfalls
Treating a continuous density as proof of validity; sorting can exist without a density jump.
Lessons and concepts
Claims and their sources
Regression discontinuity identification relies on continuity of counterfactual outcomes in the running variable, which may fail if agents can manipulate it, and a test of continuity of the running-variable density at the cutoff checks for this.