Skip to content
ML Academy

Statistics

Interview question

  • seniorlevel
  • 12 minsuggested time
  • 3Data scientist, Data analyst, ML engineer

An A/B test randomises logged-in users, but the analyst computes the standard error of revenue per session with every session treated as an independent row: 10,000 users, about 5 sessions each, a z-score of 2.58 (p = 0.01). Within a user, sessions are strongly correlated (intra-user correlation about 0.3, hypothetical). A colleague says the sample is 50,000 rows, so the result is solid. What do you say, and how do you analyse it properly?

Tests: Unit of randomisation versus unit of analysis; clustered observations; design effect; choosing a valid variance estimator (user-level aggregation, cluster-robust or delta method).

Sign in to practise: answers are saved before model answers are shown.