Statistics, experimentation, causal reasoning · insight-1009

Keep a long-term holdback to measure what a year of shipped wins really added

Situation
A quarterly deck sums the lifts of 14 individually significant experiments and claims +9%, while the dashboard moved +2%.
Action
Hold a small random slice of users (a few percent) on the old experience for the whole period and compare it with everyone else at the end. Report that delta as the cumulative effect and the sum of launch estimates only as an ex-ante figure.
Reason
Significant estimates from noisy tests tend to exaggerate effects, and stacked changes interact; only a concurrent randomised comparison of old versus new product is unbiased.
Boundary
Mature products with a steady stream of incremental experiments.

Symptom or risk

Launch-time estimates are inflated by selection, overlap the same improvement, fade after novelty and interact, so the sum does not reproduce.

Changes that cannot coexist with the old version for engineering or legal reasons.

Holdback users get an older product, and maintaining the old paths costs engineering time.

Letting shared services or personalisation leak new behaviour into the holdback.

In low-power designs, statistically significant estimates tend to overstate the true effect (Type M error).
Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors