Neural networks, deep learning, NLP · insight-1401
Fix the total step budget before you choose a decaying learning-rate schedule
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You pick a cosine or linear decay for a run and later decide to train "a few more epochs" because validation is still improving.
Action
Decide the number of optimizer steps first, derive the schedule from it, and log steps rather than epochs. To train longer, launch a new run with a new horizon, or plan restarts from the start.
Reason
A decay schedule is a function of progress through a fixed budget. Warm restarts were proposed precisely to reset the rate periodically and recover fast early progress.
Boundary
Cosine, linear and polynomial decays, and one-cycle policies, in any framework.
Symptom or risk
The schedule was shaped for the original horizon, so the extra epochs run at a learning rate already near zero. The gain looks like "more training does not help" when the schedule, not the model, stopped learning.
Exceptions
Constant-rate or plateau-based schedules have no horizon; they can be extended freely.
Trade-offs
Fixing the budget early forces a compute decision before you have seen the learning curve.
Pitfalls
Resuming from a checkpoint with a freshly built scheduler silently restarts the decay from step zero.
Lessons and concepts
Claims and their sources
SGDR periodically restarts the learning rate in stochastic gradient descent to get faster early progress when training deep networks.