Skip to content
ML Academy

Classical ML

Interview question

  • midlevel
  • 8 minsuggested time
  • 2Data scientist, ML engineer

A model predicts whether a customer will rebook within 90 days. The table has 10,000 rows from 2,000 customers (five bookings each, on average). A teammate evaluates it with ordinary shuffled 5-fold cross-validation and reports AUC 0.91; in a production pilot on new customers the AUC is 0.74. What happened, and what would you change?

Tests: Group leakage in cross-validation, why identity-like features make random splits optimistic, and choosing a split that mirrors the deployment unit.

day 15

Sign in to practise: answers are saved before model answers are shown.