Neural networks, deep learning, NLP · insight-1435

Do not stop at the first rise in validation loss without checking a longer horizon

Situation
An over-parameterised network's validation error rises in mid-training, then, with patience set to five epochs, training halts and a smaller model is chosen instead.
Action
Run one or two reference jobs with patience disabled to the full schedule, compare them with the early-stopped model on the same validation set, and set patience from what you see.
Reason
Nakkiran and colleagues report that test performance can worsen and then improve as model size or training time increases, so early stopping can end a run before a better region.
Boundary
Large, over-parameterised networks trained with noisy labels or to near-zero training loss.

Symptom or risk

In some regimes error first rises and then falls again with more training or larger models, so a short patience rule can lock in a worse optimum.

Small, well-regularised models where the curve is a clean U shape.

Extra full-length runs; justify them by the stakes of the decision.

Treating the effect as a licence to train indefinitely when the final epochs are simply overfitting.

Nakkiran et al. (arXiv preprint) report that test performance can get worse and then better as model size grows, and that the same pattern can appear as training time grows.
Deep Double Descent: Where Bigger Models and More Data Hurt