Statistics, experimentation, causal reasoning · insight-0008
Only segment experiment results by attributes the treatment cannot change
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist, Data analyst
- Industries
- e-commerce, aviation, saas
Summary: situation, action, reason, boundary
Situation
A readout splits users by "saw the new fare-comparison widget", "completed search" or another behaviour measured after assignment.
Action
Segment only on pre-assignment attributes such as country, device at first exposure or tenure at start. For any segment that might be affected, run a sample-ratio check within the segment and discard segment results if it fails.
Reason
A post-treatment segment is a collider-like selection: composition changes, not behaviour changes, can drive the within-segment difference.
Boundary
Experiment slicing, funnel analysis and any "treated users who did X" comparison.
Symptom or risk
Treatment moves users between segments, so both segments can improve while the total does not move, and segment results are biased.
Exceptions
Triggered analysis where the trigger condition is computed identically for both arms, including for control users who would have seen the change.
Trade-offs
Pre-assignment segments are less intuitive for product teams but give interpretable effects.
Pitfalls
Defining segments from end-of-test values of attributes that drift during the test, such as loyalty tier.
Lessons and concepts
Claims and their sources
When the treatment changes which users fall into a segment, segment-level movements can reverse or disappear in aggregate (Simpson's paradox); a per-segment sample ratio test detects this.