Classical ML, validation, metrics, interpretation, debugging · insight-1139draft — in review
Score early-stopped boosting folds on data the stopping rule never saw
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Inside cross-validation you pass each validation fold to the booster as its early-stopping set and report that fold's metric.
Action
Split a stopping set off each training fold, or use nested folds, and score the outer fold only after the rounds are fixed; for the final model use the median best round count.
Reason
The best iteration is a hyperparameter fitted on the evaluation set, and a score used for selection is biased as an estimate of generalization.
Boundary
LightGBM, XGBoost and CatBoost with early stopping inside cross-validation or on a single validation split.
Symptom or risk
The number of rounds was chosen to minimize error on that very fold, so the reported score is selected and optimistic, like any tuned hyperparameter.
Exceptions
When there are thousands of validation rows and a smooth loss curve, the bias is small; still prefer a clean outer set for reporting.
Trade-offs
Less data for fitting each fold in exchange for an honest score.
Pitfalls
Early stopping on several validation sets or metrics, where any one stalling can end training.
Lessons and concepts
Claims and their sources
LightGBM's early_stopping_round stops training when a validation metric has not improved for that many rounds, with first_metric_only controlling which metrics count.
Tuning hyperparameters on the same cross-validation estimate used for reporting yields an optimistically biased performance estimate.