Statistics, experimentation, causal reasoning · insight-1033
Plan power for a ratio metric from numerator-denominator covariance by customer, not the spread across rows
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist
- Industries
- aviation, e-commerce, saas
Summary: situation, action, reason, boundary
Situation
A test targets revenue per booked passenger, and the sample size was computed from the standard deviation of the ratio across bookings.
Action
Aggregate to the randomisation unit, estimate the variance of the ratio of user-level sums with the delta method (variance of the numerator, of the denominator, and their covariance), or bootstrap customers on historical data, and size the test from that.
Reason
The metric is a ratio of two averages over randomisation units, and its variance includes a covariance term that row-level formulas ignore.
Boundary
Revenue per booking, click-through per session, items per order.
Symptom or risk
Randomisation is by customer, so bookings of one customer are correlated and the numerator and denominator co-move; the sizing is wrong in either direction.
Exceptions
Metrics with one observation per randomisation unit.
Trade-offs
Needs historical user-level data to estimate the covariance.
Pitfalls
Choosing between the ratio of totals and the mean of user-level ratios after seeing which is significant; they are different estimands.
Lessons and concepts
Claims and their sources
When randomising by user but computing page-level metrics, observations are not independent, and the metric can be written as a ratio of two averages of randomisation-unit quantities whose variance follows from the delta method.