Statistics, experimentation, causal reasoning · insight-1002

Check how well the pre-period predicts the metric before counting on CUPED to save traffic

Situation
A test plan halves the duration "because we use CUPED", but the metric is first-visit conversion or the population is mostly new customers.
Action
Measure the correlation between the pre-period covariate and the in-test metric on past data, per segment. Expect a variance reduction near 1 minus the squared correlation and size the test with that factor. Where no history exists, adjust only for attributes known at assignment, such as market or device.
Reason
CUPED removes the part of the metric that pre-experiment data predict; without predictability there is nothing to remove.
Boundary
User-level metrics with a usable history, such as repeat bookers.

Symptom or risk

The covariate barely correlates with the outcome, or does not exist for new users, so variance hardly falls and the shortened test is underpowered.

Strongly persistent metrics, for example pre-period spend for loyalty members, where the gain can approach the reported order of magnitude.

Extra pipeline complexity for a gain that may be small.

Using a covariate measured after assignment, which can absorb part of the treatment effect.

Hypothetical correlations: 0.3 cuts variance by about 9%, 0.7 by about 51%.

CUPED uses pre-experiment data to reduce metric variability, and the authors report roughly 50% variance reduction on Bing metrics.
Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data