Skip to content
ML Academy

Experimentation and causality

Interview question

  • stafflevel
  • 25 minsuggested time
  • 2Data scientist, Data analyst

Your company runs about 200 A/B tests a year. The product organisation reports that 25 'winning' tests shipped last year, with measured lifts that sum to +25% in the key metric. The company's metric rose 6%. Finance asks whether the experimentation programme is worth its cost. As the staff data scientist, what do you tell them and what changes in the programme?

Tests: Program-level thinking about experimentation: false discovery rate, winner's curse, interaction and decay, long-term holdouts, and incentives.

day 4day 5day 10

Sign in to practise: answers are saved before model answers are shown.