Neural networks, deep learning, NLP · insight-0414
Retune the learning rate whenever you change the batch size
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You move training to a bigger GPU and raise the batch size from 64 to 512 to finish faster, keeping the old learning rate.
Action
Treat batch size and learning rate as one setting. As a starting point scale the learning rate with the batch size and add a warmup, then confirm with a short range test.
Reason
Goyal et al. trained ImageNet with very large minibatches without accuracy loss by scaling the learning rate linearly with minibatch size and adding warmup.
Boundary
SGD-style training where you change the batch size by large factors.
Symptom or risk
With fewer, less noisy updates per epoch, the old learning rate under-trains; results get worse and the hardware change is blamed on the model.
Exceptions
The linear rule breaks down at very large batches and is less reliable for adaptive optimizers; validate instead of assuming.
Trade-offs
Larger batches use hardware better but give fewer updates per epoch and may need more epochs.
Pitfalls
Comparing two models trained at different batch sizes with the same learning rate and calling the difference architectural.
Lessons and concepts
Claims and their sources
Goyal et al. used a linear scaling rule for learning rate vs minibatch size plus a warmup scheme, reporting no accuracy loss up to minibatch 8,192 on ImageNet.