Statistics, experimentation, causal reasoning · insight-0005

Name one primary metric and correct for multiplicity across the rest of the scorecard

Situation
An experiment scorecard shows 20 or more metrics and the readout highlights whichever ones moved.
Action
Pre-register one primary decision metric and a small set of guardrails. Treat other metrics as diagnostics; when you must claim effects on many, control the false discovery rate (Benjamini-Hochberg) or the family-wise rate (Bonferroni or Holm).
Reason
The 5% error rate applies per test. Picking winners after the fact turns many honest tests into one dishonest claim.
Boundary
Experiment scorecards, segment breakdowns, metric screens and model comparisons on one test set.

Symptom or risk

With 20 independent metrics at the 5% level, the chance of at least one false "win" is about 64% even when nothing changed.

Exploratory analysis meant to generate hypotheses, labelled as such and followed by a confirmatory test.

Bonferroni protects strongly but loses power; FDR control keeps more power and accepts that a known share of claimed discoveries are false.

Applying a correction to the metrics you report while silently dropping the ones you looked at.

The Benjamini-Hochberg procedure controls the expected proportion of false rejections (the false discovery rate) for independent test statistics.
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing