Classical ML, validation, metrics, interpretation, debugging · insight-0168
Give early stopping its own validation split and never point it at the test set
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You train LightGBM, XGBoost or a neural network with early stopping and pass the test set as the evaluation set.
Action
Carve an early-stopping validation set from training data (or use inner folds), keep the test set untouched, and record the chosen number of rounds.
Reason
Early stopping selects a hyperparameter (the number of iterations) by watching a metric; whatever data it watches is no longer independent.
Boundary
Gradient boosting, neural networks and any iterative learner with a stop-on-plateau rule.
Symptom or risk
The number of rounds is selected on the test set, so the reported test score is optimistically biased.
Exceptions
None for the final estimate; for quick exploration, label such results as validation scores.
Trade-offs
An extra split reduces training data; refitting on all training data with a fixed round count recovers it but loses the stopping signal.
Pitfalls
Passing several evaluation metrics without noticing that any one of them can trigger stopping (or only the first, with first_metric_only), so the round count is not tuned for the metric you report.
Lessons and concepts
Claims and their sources
LightGBM's early_stopping_round stops training when a validation metric has not improved for the given rounds; first_metric_only restricts it to the first metric.