Statistics, experimentation, causal reasoning · insight-0001
Size an A/B test from the smallest effect you would act on, not the effect you hope to see
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- aviation, e-commerce, cross-industry
Summary: situation, action, reason, boundary
Situation
A product owner asks how long to run a booking-page test and offers the uplift they expect, or no target at all.
Action
Agree the minimum detectable effect (MDE) as the smallest change that would justify shipping. Compute sample size from that MDE, the baseline rate, the significance level and the target power, then convert it to days of traffic.
Reason
Required sample size scales with the inverse square of the effect, so halving the MDE roughly quadruples the traffic needed. Anchoring on the decision threshold makes the test answer the business question.
Boundary
Fixed-horizon tests on conversion, revenue per visitor or other user-level metrics.
Symptom or risk
The test is sized for an optimistic effect, ends underpowered, and a realistic but valuable uplift is reported as "no effect".
Exceptions
When traffic can never reach the required size, switch to a larger unit of change, a stronger metric proxy or a non-experimental design with stated limits.
Trade-offs
A smaller MDE protects against missing modest wins but costs weeks of traffic and delays other tests.
Pitfalls
Computing power after the test from the observed effect; it adds nothing beyond the p-value.
Example
Hypothetical: 4% baseline conversion, 0.2 percentage-point MDE, 5% two-sided level and 80% power need about 154,000 sessions per arm.