Statistics, experimentation, causal reasoning · insight-0018

Decide how to treat revenue outliers before the test starts, then apply it to both arms

Situation
Revenue per user in an experiment swings because a few corporate group bookings land in one arm.
Action
Pre-specify a capping rule (for example, cap at a high percentile computed on pooled pre-period data) or a log-type transformation, and report the uncapped result as a sensitivity check. Investigate whether extremes are bots, data errors or real customers.
Reason
Heavy tails inflate variance; capping trades a small bias for a large gain in precision, and deciding it in advance prevents choosing the rule that produces significance.
Boundary
Revenue, order value, session length and other heavy-tailed metrics.

Symptom or risk

A handful of extreme values dominate the mean and the variance, so the test is underpowered and its sign can flip with one booking.

If the change targets high-value customers, capping removes the effect you want to measure; analyse that segment directly.

Capped metrics are easier to detect but harder to translate into total revenue.

Computing the cap separately per arm, which lets the treatment change the cap.

Outliers can dominate experiment metrics; capping values or transforming the metric (for example with a log) reduces their impact at some cost to interpretability.
A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments