Classical ML

Interview question

midlevel
10 minsuggested time
2Data scientist, ML engineer

A teammate classifies 100 labelled claims as fraudulent or not using 2,000 candidate features extracted from claim text and metadata. To keep the model small they keep the 20 features most associated with the label (an F-test on all 100 rows), then run 5-fold cross-validation of a logistic regression on those 20 and report AUC 0.93. Nothing in the code touches the test folds after that. Is 0.93 credible? How would you prove it either way, and what would you change?

Tests: Recognising selection leakage (preprocessing that uses the labels before the validation split), knowing how large the effect is when features greatly outnumber rows, and using a label-permutation check and a pipeline that refits the selection inside every fold.

day 17 day 15 day 9