Neural networks, deep learning, NLP · insight-0413
Run a learning-rate range test before tuning anything else
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You start training a new architecture or dataset and are about to grid-search learning rate together with other hyperparameters.
Action
Train for a short run while increasing the learning rate steadily from very small to large (linearly in Smith's description, often exponentially in library implementations), plot loss against learning rate, and pick a value below the point where loss stops falling and starts to rise.
Reason
The range test finds the region where the loss decreases fastest in minutes, so later tuning starts from a sensible scale.
Boundary
New models or datasets trained with SGD or Adam-family optimizers.
Symptom or risk
A wrong learning rate dominates every other effect: you may conclude a model or feature does not work when the optimizer was simply too slow or unstable.
Exceptions
Fine-tuning large pretrained models usually uses small, well-documented learning rates; a range test is less informative.
Trade-offs
The curve is noisy and depends on batch size; rerun it if you change batch size or optimizer.
Pitfalls
Picking the learning rate at the minimum of the curve, which is often already too high.
Lessons and concepts
Claims and their sources
Smith describes estimating reasonable learning-rate bounds by increasing the learning rate linearly during a short run.