Classical ML, validation, metrics, interpretation, debugging · insight-1181
Count the minority events in each validation fold before trusting a rare-event metric
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- banking, manufacturing, aviation
Summary: situation, action, reason, boundary
Situation
An event with about 1% prevalence is validated with 5-fold cross-validation on 3,000 rows, which leaves about 30 events in total and 6 per fold.
Action
Print events per fold, use repeated stratified cross-validation and report the spread of the metric, compare candidates on identical folds with paired differences, and say plainly when the event count cannot separate models.
Reason
Metric precision depends on the number of positives, not on the number of rows.
Boundary
Fraud, no-show, failure and default models with small event counts.
Symptom or risk
Fold recall, precision and PR-AUC rest on a handful of cases; one case moves recall by about 17 points, model comparisons are noise, and a fold with no events leaves AUC undefined.
Exceptions
Plenty of events per fold, where ordinary stratified folds are enough.
Trade-offs
Repetition costs compute, and stratification can make fold scores look more alike than they should.
Pitfalls
Reading a 0.02 gap in PR-AUC between two models evaluated on 30 events as a winner.
Lessons and concepts
Claims and their sources
StratifiedKFold keeps approximately the same class percentages in each fold; plain KFold on 45 versus 5 samples left a test fold with no samples of the rare class, and RepeatedStratifiedKFold repeats stratified folds with different randomization.