Statistics, experimentation, causal reasoning · insight-1007
Analyse a campaign by assignment first, and read engaged-only numbers with their assumptions attached
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- aviation, retail, telecom
Summary: situation, action, reason, boundary
Situation
An email offer is randomised; only 20% open it, and the analyst compares openers in the treatment arm with everyone in control.
Action
Report the intention-to-treat effect (all assigned versus all control) as primary. For an effect among those who engage, scale by take-up only under stated assumptions: assignment affects outcomes solely through exposure, and non-openers are unaffected.
Reason
Assignment is random while exposure is not; ITT preserves the randomisation and per-protocol estimates need untestable adjustments.
Boundary
Campaigns, onboarding flows and any test where exposure depends on user behaviour.
Symptom or risk
Openers are self-selected and more engaged, so the comparison measures who opens, not what the offer does.
Exceptions
Full-compliance designs such as server-side price changes.
Trade-offs
ITT is diluted by non-takers; it answers the rollout question, not the "if they engage" question.
Pitfalls
Dropping non-openers from both arms, which breaks the arm comparison.
Example
Hypothetical: ITT lift 0.3 percentage points with 20% take-up implies about 1.5 points among takers under the stated assumptions.
Lessons and concepts
Claims and their sources
The intention-to-treat comparison is not guaranteed to estimate effectiveness, and as-treated or per-protocol analyses adjusted by weighting rely on untestable assumptions.