Neural networks, deep learning, NLP · insight-0423
Early-stop neural training on the decision metric and restore the best checkpoint
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- telecom
Summary: situation, action, reason, boundary
Situation
A churn network uses early stopping on validation loss with default patience, and the final weights are whatever the last epoch produced.
Action
Evaluate the decision metric every epoch (or every N steps), stop after a patience window with a minimum improvement, save a checkpoint at each new best, and reload that checkpoint before calibration and testing.
Reason
Early stopping regularizes by choosing the number of training steps on validation data, so the selection criterion should be the quantity you will act on, and the selected weights must be the ones you keep.
Boundary
Neural training runs where evaluating the validation set regularly is affordable.
Symptom or risk
Neural validation curves are noisy and loss can keep improving while the metric the decision uses, such as recall at a fixed precision, has already peaked; the shipped weights are not the best ones seen.
Exceptions
Very noisy decision metrics on small validation sets; smooth them or select on loss and report the metric.
Trade-offs
Patience too short stops before late gains, especially with learning-rate schedules; too long wastes compute.
Pitfalls
Stopping on the metric of the last mini-batch instead of the full validation set.
Lessons and concepts
Claims and their sources
Goodfellow, Bengio and Courville (ch. 7.8) describe early stopping as regularization in which the number of training steps is a hyperparameter selected on validation data.