Skip to content
ML Academy

Experimentation and causality

Interview question

  • midlevel
  • 8 minsuggested time
  • 2Data scientist, Data analyst

Your company's experimentation platform has just been rebuilt. Before trusting it, you are asked to run 100 A/A tests (both arms see the same experience) at a 5% significance level. What do you expect to see, what result would make you suspicious, and what would you check if the platform fails the test?

Tests: Understanding the null distribution of p-values, false-positive rate calibration, and using A/A tests to validate assignment and variance estimation.

day 4day 5

Sign in to practise: answers are saved before model answers are shown.