Statistics, experimentation, causal reasoning · insight-1013

Choose the multiplicity family and error rate by what a false positive costs, and say which you used

Situation
A route-level analysis tests 80 hypotheses; one analyst uses Bonferroni, another raw p-values, a third Benjamini-Hochberg, and the reports cannot be compared.
Action
Define the family as all hypotheses behind one decision before looking. Use familywise control (Holm or Bonferroni) when one false claim is costly, and false discovery rate control when screening candidates for follow-up. Under arbitrary dependence use the conservative variant or permutation, and report raw and adjusted values.
Reason
The Benjamini-Hochberg procedure controls the false discovery rate for independent or positively dependent tests; a more conservative modification covers any dependence at a power cost.
Boundary
Route, segment and feature scans with many simultaneous tests.

Symptom or risk

Familywise correction discards real leads in a screening exercise, while raw p-values flood a launch decision with false positives.

A single pre-registered primary hypothesis, which needs no adjustment.

Stricter control raises the bar for every lead, including real ones.

Shrinking the family after seeing which tests failed.

The Benjamini-Hochberg procedure controls the false discovery rate for independent test statistics.
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
The same procedure controls the false discovery rate under positive regression dependence, and a conservative modification controls it under any dependence.
The control of the false discovery rate in multiple testing under dependency