Skip to content
ML Academy

Classical ML

Interview question

  • seniorlevel
  • 12 minsuggested time
  • 2Data scientist, ML engineer

A team tunes a gradient-boosting model with a random search of 200 configurations on one validation set and reports the best validation AUC (0.83) as the expected production performance. The best and median configurations differ by only about 0.03 AUC, and the validation noise on AUC is about 0.01 per configuration (hypothetical). A reviewer objects. Who is right, and what should the team report?

Tests: Selection bias from choosing the maximum of noisy estimates, validation-set reuse, nested validation and reporting an honest generalisation estimate.

day 15day 4

Sign in to practise: answers are saved before model answers are shown.