Skip to content
ML Academy

Experimentation and causalitydraft — in review

Interview question

  • stafflevel
  • 25 minsuggested time
  • 2Data scientist, Data analyst

Over a year your product organisation ran 300 experiments and shipped the 25 "wins" (significant at 0.05, positive). The shipped tests claimed an average lift of 1.3% each, and the sum of claimed lifts is above 30%. Yet the company's core metric grew about 6%. Leadership asks whether the experiment program is broken and what you would change. (Hypothetical program figures.)

Tests: Winner's curse and selection on noisy estimates, empirical Bayes shrinkage, interaction and decay effects, and designing an experimentation program that gives trustworthy aggregate numbers.

day 4day 5day 16

Sign in to practise: answers are saved before model answers are shown.