Experimentation and causalitydraft — in review

Interview question

seniorlevel
12 minsuggested time
1Data scientist

You randomise 50/50 on whether users are offered a new "trip planner" flow. Only 30% of users in the treatment arm actually open it; control users cannot access it. Booking conversion is 5.0% in control and 5.6% in treatment (hypothetical). A PM says "users who opened the planner convert at 9%, so it works". What is the effect of the offer, what is the effect of using the planner, and what can you honestly claim?

Tests: Intent-to-treat vs complier average causal effect, selection bias in comparing users, and the assumptions needed to scale the ITT.

day 5