Experimentation and causality

Interview question

seniorlevel
12 minsuggested time
2Data scientist, Data analyst

A new in-app 'price alert' feature is rolled out in an A/B test: 50,000 users get access to the feature and 50,000 do not (hypothetical). Only 30% of users with access ever turn the feature on. Booking conversion is 5.0% in control and 5.6% in the treatment group. A teammate compares users who turned the feature on with those who did not (in the treatment group) and gets +4 points. Which number do you report, and what does each mean?

Tests: Intention-to-treat versus treatment-on-the-treated, complier average causal effect, selection into usage, and uncertainty propagated to the scaled estimate.

day 5