Statistics, experimentation, causal reasoning · insight-0010
Use a pre-period covariate (CUPED-style adjustment) to shrink variance before buying more traffic
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- e-commerce, saas, cross-industry
Summary: situation, action, reason, boundary
Situation
An experiment on revenue per user needs more weeks than the roadmap allows, and the metric has a strong user-level history.
Action
Adjust each user's outcome with the same metric measured before assignment, using a regression coefficient estimated on pooled data. Compare adjusted and unadjusted estimates; the point estimate should agree in expectation while the interval narrows.
Reason
The pre-period value is independent of treatment under randomization, so removing the part of variance it explains keeps the estimate unbiased; variance shrinks roughly by the squared correlation between pre and post values.
Boundary
Returning users, B2B accounts and stores with stable history.
Symptom or risk
The team either runs underpowered tests or waits months, while much of the metric's variance is predictable from each user's past behaviour.
Exceptions
New users without history gain little; covariates measured after assignment can be affected by treatment and must not be used.
Trade-offs
Extra pipeline complexity and explanation effort in exchange for shorter tests.
Pitfalls
Choosing the covariate after looking at which one gives the smallest p-value.
Lessons and concepts
Claims and their sources
CUPED uses pre-experiment data to reduce metric variance; on Bing it reduced variance by about 50%, equivalent to the same power with half the users or half the duration.