Statistics, experimentation, causal reasoning · insight-0011
Match the variance formula to the randomization unit when metrics are computed per page or per session
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- e-commerce, saas, aviation
Summary: situation, action, reason, boundary
Situation
Users are randomized, but the metric is click-through per page view or conversion per search, and the analysis treats each page as an independent observation.
Action
Write the metric as a ratio of per-user sums (total clicks over total page views) and use the delta method or a user-level bootstrap for its variance. Run an A/A test to confirm the false-positive rate is near nominal.
Reason
Independence holds across randomization units, not within them. The delta method treats the metric as a ratio of two averages of independent user-level quantities.
Boundary
Session, page, search or booking-level metrics in user-randomized tests; store-level randomization with transaction-level metrics.
Symptom or risk
Pages from the same user are correlated, so the naive standard error is too small and false positives multiply.
Exceptions
When randomization and analysis units coincide (user-level metric, user randomization), the standard formula is fine.
Trade-offs
Correct intervals are wider and reveal that some past "wins" were noise.
Pitfalls
Randomizing at a finer level than the analysis unit, which puts one user in both arms.
Lessons and concepts
Claims and their sources
When the randomization unit is a cluster of analysis units, observations are not i.i.d.; writing the metric as a ratio of randomization-unit averages lets the delta method estimate its variance.