Statistics, experimentation, causal reasoning · insight-0003
Do not stop a fixed-horizon A/B test the first time the dashboard turns significant
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- e-commerce, saas, aviation
Summary: situation, action, reason, boundary
Situation
Stakeholders check the experiment dashboard daily and ask to ship as soon as the p-value drops below 0.05.
Action
Either commit to the planned sample size and read the result once, or adopt a design built for monitoring, such as a group-sequential plan with a few pre-set looks or always-valid p-values. Hide significance markers on interim views of fixed-horizon tests.
Reason
Standard p-values assume the sample size was fixed in advance; stopping at the first crossing selects the most extreme moment of a random walk.
Boundary
Any frequentist test analysed with a fixed-sample formula.
Symptom or risk
Each look is another chance for noise to cross the threshold, so the real false-positive rate ends far above the nominal 5% and neutral changes get shipped.
Exceptions
Stopping early for harm on a guardrail metric is a safety decision and can be pre-specified with its own boundary.
Trade-offs
Sequential methods allow earlier decisions but have wider intervals at any given sample size than a fixed design.
Pitfalls
Extending a flat test "to gain power" is the same problem in the other direction.
Lessons and concepts
Claims and their sources
Continuously checking results and stopping (or extending) when significance appears increases the Type I error of a standard null-hypothesis test.
Always-valid p-values and confidence intervals remain valid when users monitor tests continuously and stop adaptively.