Skip to content
ML Academy

Introduced on day 4

Multiple comparisons

Running many tests (segments, metrics, repeated looks) so that at least one false positive becomes likely even when nothing is real.

In detail

With 20 independent tests at the 5% level and no true effects, the chance of at least one p < 0.05 is about 64%. Remedies: pre-registration, corrections (Bonferroni, Benjamini-Hochberg), sequential designs, and separate confirmation tests.