Statistics, experimentation, causal reasoning · insight-0013
Communicate a null result with the effects it rules out, not as "no impact"
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A well-run test of a new upsell banner ends with p = 0.4 and a stakeholder asks whether the feature "does nothing".
Action
Report the point estimate and confidence interval in business units, and state which effects the data exclude. If the decision needs a "no meaningful effect" claim, set the smallest effect of interest in advance and run an equivalence test (two one-sided tests).
Reason
A non-significant result may reflect low power. An interval or an equivalence test turns the null into a statement about plausible effect sizes.
Boundary
Experiments, model-change evaluations and observational comparisons used for go/no-go decisions.
Symptom or risk
"Not significant" is read as "proven zero", so a useful change is abandoned or a harmful one is called safe.
Exceptions
Exploratory screens where the only decision is whether to investigate further.
Trade-offs
Equivalence claims need more data than detecting an effect of the same size.
Pitfalls
Choosing the equivalence margin after seeing the interval.
Example
Hypothetical: uplift +0.3% with a 95% interval from -0.4% to +1.0% (consistent with p near 0.4) rules out losses beyond about 0.4% but not a gain of nearly 1%.
Lessons and concepts
Claims and their sources
Interpreting P > 0.05 as showing no effect is a common misinterpretation of significance tests.
Equivalence testing (TOST) uses bounds set by the smallest effect size of interest and can reject effects large enough to matter, which a non-significant test cannot.