Statistics, experimentation, causal reasoning · insight-0002

Discount significant wins from underpowered tests because their effect sizes are exaggerated

Situation
A small or short test returns a statistically significant uplift that is several times larger than past changes of the same kind.
Action
Before celebrating, estimate the design's power for a plausible effect taken from past experiments. If power is low, treat the estimate as an upper-biased signal, report the wide interval, and replicate or extend under a pre-registered plan before booking the gain.
Reason
When only significant results are reported, a noisy design can pass the threshold only by overestimating the effect; the lower the power, the larger the expected exaggeration.
Boundary
Small segments, low-traffic routes, short tests and any metric with heavy-tailed variance.

Symptom or risk

The team books the inflated uplift into a forecast, and later results regress toward a much smaller effect or even the opposite sign.

Well-powered designs, where significant estimates are close to unbiased.

Replication delays a launch; the alternative is planning around a number that will not materialize.

Using the observed effect to compute power, which makes every significant result look well powered.

In low-power designs, statistically significant estimates tend to overstate the true effect (Type M error, the exaggeration ratio) and can have the wrong sign (Type S error); the problem grows as power drops well below 0.5.
Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors