Statistics, experimentation, causal reasoning · insight-1015

Write down exclusions, covariates and the stopping rule before unblinding, because each free choice is a hidden test

Situation
After a flat result, an analyst tries excluding bot-like sessions, adding a covariate, trimming the top 1%, and switching to log revenue, and reports the variant with p = 0.03.
Action
Freeze a one-page analysis plan before opening the outcome data: primary metric, population, exclusions, transformations, covariates and stopping rule. Label anything else as exploratory with the count of variants tried, and archive the first run.
Reason
Flexibility in choosing outcomes, sample size, covariates and reported conditions inflates the false-positive rate well above the nominal level, even without bad intent.
Boundary
Any analysis where the data are seen before the pipeline is fixed.

Symptom or risk

Every flexible choice is another chance to cross 0.05, and the reported p-value does not reflect the paths that were tried.

Pure exploration, provided it is not presented as confirmation.

Less room to react to a surprising data problem; handle by documented, blind data-quality rules.

A plan so vague that every choice still fits it.

Flexibility in choosing dependent variables, sample size, covariates and which conditions to report substantially raises the false-positive rate above the nominal 5%, as shown by simulation and two experiments.
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant