Statistics, experimentation, causal reasoning · insight-0016

At very large sample sizes, judge the effect against the action threshold, not the p-value

Situation
A test with millions of sessions reports p < 0.0001 for a 0.02% conversion change, or an observational query over a year of bookings finds every coefficient significant.
Action
State the effect with its interval in money, bookings or minutes, and compare it with the cost of shipping and maintaining the change. Decide on the practical threshold agreed before the test.
Reason
Standard errors shrink with sample size, so any non-zero difference eventually becomes significant; significance stops carrying information about importance.
Boundary
High-traffic experiments, large-scale observational analyses and model comparisons on large test sets.

Symptom or risk

Tiny effects get treated as important, consuming engineering time and adding complexity that never pays back.

Small relative changes on enormous bases can still be worth shipping; compute the absolute value before dismissing them.

A practical threshold requires an explicit value judgment that some stakeholders prefer to avoid.

Reporting only relative lift, which hides whether the absolute change is material.

A statistically significant result does not by itself indicate an important or large effect.
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations