Experimentation and causality

Interview question

seniorlevel
10 minsuggested time
3Data scientist, Data analyst, ML engineer

An experiment configured for a 50/50 split ends with 50,600 users in control and 49,400 in treatment. Treatment shows a significant +2% revenue lift. Do you trust it?

Tests: Detecting sample ratio mismatch (SRM), understanding its causes, and refusing to interpret effects until assignment integrity is restored.

day 4 day 5