Skip to content
ML Academy

Experimentation and causalitydraft — in review

Interview question

  • seniorlevel
  • 12 minsuggested time
  • 2Data scientist, Data analyst

A pricing-page test randomised 40,000 users (20,000 per arm). The analyst compared revenue per session across about 160,000 sessions with a two-sample test and got p = 0.03. Sessions per user average 4 and the correlation between two sessions of the same user is about 0.3 (hypothetical figures). Do you ship? How would you analyse this test correctly?

Tests: Unit of randomisation vs unit of analysis, intra-user correlation, design effect, and variance estimation for ratio metrics (clustered standard errors or the delta method).

day 4day 5

Sign in to practise: answers are saved before model answers are shown.