Neural networks, deep learning, NLP · insight-1431
Get a model that overfits the full training set before you add regularisation
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A new network underperforms and you immediately add dropout, weight decay and augmentation to it all at once.
Action
First enlarge the model or train longer until training loss is near its floor on the full set. Then add one regulariser at a time, each with its validation effect and seed spread.
Reason
A model that cannot overfit signals a problem in the data, loss or optimisation that regularisation would hide, and one-at-a-time changes keep attributions clean.
Boundary
New architectures, datasets and pipelines.
Symptom or risk
If it cannot fit the training data, regularisation only deepens the underfit, and you cannot tell whether the problem is capacity, optimisation or a bug.
Exceptions
Very noisy targets where training loss never approaches zero; use a noise-floor baseline as the goal.
Trade-offs
Takes more runs than turning everything on; mitigated by small subsets.
Pitfalls
Calling a model "overfit" because validation is bad when training loss is also high.
Lessons and concepts
Claims and their sources
Karpathy's recipe has the practitioner first overfit, with a model large enough to drive training loss down, and only then regularise.