Engineering, production, governance, cost, communication · insight-0294
Report a null result with its interval and detectable effect, not as 'no effect'
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- e-commerce, saas, retail
Summary: situation, action, reason, boundary
Situation
A test of a new recommendation model shows no statistically significant revenue change, and the summary says 'the model does not work'.
Action
State the estimated effect with its confidence interval, the minimum detectable effect the test could find, and whether the interval excludes the smallest effect that would matter; then recommend stop, extend or redesign.
Reason
A non-significant p-value does not show that there is no effect; the interval shows which effects remain plausible.
Boundary
Experiments, model launches, backtests comparing models, observational analyses.
Symptom or risk
A non-significant result may come from an underpowered test; declaring no effect can kill a useful change or justify a harmful one.
Exceptions
When the interval is narrow and excludes all business-relevant effects, 'no meaningful effect' is a fair summary.
Trade-offs
Nuanced messages are harder to act on; end with a clear recommendation.
Pitfalls
Running the test longer only when results are null, which biases future comparisons.
Lessons and concepts
Claims and their sources
A non-significant result does not imply that the null hypothesis is true or that the effect is absent.