Statistics, experimentation, causal reasoning · insight-0009

Plot the treatment effect by day of exposure before trusting a short-term uplift

Situation
A new seat-map design or prompt shows a strong uplift in a two-week test and the team wants to extrapolate it to the annual plan.
Action
Break the effect down by day since first exposure or by visit number, and compare first-visit with later-visit effects. If the curve is still moving at the end, extend the test, keep a long-term holdout, or report the trend rather than a single number.
Reason
A short test averages over a transient period; the business decision depends on the steady state.
Boundary
Visible UI changes, notifications, recommendations and anything users must learn or that adapts to them.

Symptom or risk

Users react to novelty and the effect fades, or the effect grows as users learn, so the two-week average misstates the long-run impact.

Changes users cannot perceive, such as latency or backend ranking tweaks, are less prone to novelty effects.

Longer tests and holdouts cost traffic and delay learning; they also accumulate cookie churn and survivorship among returning users.

Comparing calendar days instead of exposure days, which mixes new and returning users.

Novelty effects (an early effect that fades) and primacy effects (an effect that grows as users or models adapt) can be detected by segmenting the treatment effect by day of experiment or by user visit.
A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments