Statistics, experimentation, causal reasoning · insight-1035

Estimate your program's prior win rate before interpreting a single significant result

Situation
Suppose only about one idea in ten has delivered in your experimentation program, and a new test reaches p = 0.04.
Action
Estimate the share of past tests that were real wins from replications or holdbacks, combine it with the test's power and threshold to compute the false-finding risk, and require stronger evidence or replication for surprising claims.
Reason
The chance that a significant result is false depends on the prior probability of a real effect and on power, not only on the p-value threshold.
Boundary
Mature experimentation programs with an archive of past results.

Symptom or risk

A significant result from a low-prior program is read as 96% likely to be real, when a large share of such results are false findings.

Hypotheses backed by mechanism and prior replications, which deserve a higher prior.

Prior estimates are themselves uncertain, so show a range.

Treating the false-finding risk as a property of one test instead of the program.

Hypothetical: a 10% prior, 80% power and a 5% threshold give 0.05 x 0.9 / (0.05 x 0.9 + 0.8 x 0.1) = 36% false findings among significant results.

Claiming a discovery at p = 0.05 gives a wrong conclusion at least 30% of the time under the author's assumptions, and more when experiments are underpowered; the result depends on the assumed prior probability of a real effect.
An investigation of the false discovery rate and the misinterpretation of p-values