Statistics, experimentation, causal reasoning · insight-0006
Treat a winning segment found after a flat test as a hypothesis for the next test
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- e-commerce, saas, cross-industry
Summary: situation, action, reason, boundary
Situation
The overall result is flat, so the analyst slices by country, device, fare class and tenure until one subgroup shows a significant uplift.
Action
Report the overall result as the decision. Log the segment finding with the number of slices examined, and confirm it in a new test pre-registered on that segment.
Reason
Recursive slicing is a multiple-testing search; with enough cuts something always crosses the threshold. A fresh sample removes the selection.
Boundary
Post-hoc subgroup analysis in experiments and in observational dashboards.
Symptom or risk
The "winning segment" is very likely noise, and a feature gets targeted at a subgroup where it does nothing.
Exceptions
Segments specified before launch with a plausible mechanism, analysed with a multiplicity correction.
Trade-offs
A confirmatory test costs time but prevents a targeting rollout built on chance.
Pitfalls
Defining segments with behaviour the treatment can change, which adds Simpson's-paradox distortions on top of multiplicity.
Lessons and concepts
Claims and their sources
Recursively segmenting users until a significant movement appears is a multiple-testing problem that inflates Type I error; corrections such as Bonferroni should be applied to segment analyses.