Statistics, experimentation, causal reasoning · insight-1006
Analyse only users who could see the change, then dilute the result before forecasting the company-wide effect
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- aviation, e-commerce, saas
Summary: situation, action, reason, boundary
Situation
A seat-map change reaches 6% of sessions; the all-user test is flat, while a triggered analysis shows +3% on the page.
Action
Define the trigger from logged exposure, never from outcomes, and check sample ratio on the triggered population. Estimate the effect on triggered users, then scale by the triggered share (with ratio metrics handled explicitly) to get the overall effect.
Reason
Unexposed users add noise and no signal; the overall effect is the triggered effect weighted by the share of the population it touches.
Boundary
Features on a minority path such as seat maps, rebooking, baggage upsell.
Symptom or risk
Either a real local effect is buried in noise from unaffected users, or a +3% triggered result is booked as +3% company-wide.
Exceptions
Changes that alter behaviour before the trigger point, where triggered-only analysis misses the effect.
Trade-offs
Extra logging for counterfactual exposure in the control arm.
Pitfalls
Triggering on a post-treatment action such as a click, which selects different people in each arm.
Example
Hypothetical: 6% reach x +3% on those users is about +0.18% overall, before ratio-metric and baseline adjustments.
Lessons and concepts
Claims and their sources
Restricting analysis to triggered users and then diluting the effect to the whole population is common practice, but the dilution formulas are often applied incorrectly, especially for ratio metrics.