Neural networks, deep learning, NLP · insight-1407
Estimate the gradient noise scale to see when a larger batch has stopped helping
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Training is stable at your current batch size and you are deciding whether to double it, but a full sweep is too expensive.
Action
Log gradient norms at two batch sizes, estimate the noise scale from them, and treat batches well below it as worth enlarging and batches well above it as wasted compute. Re-estimate during training.
Reason
McCandlish and colleagues show a simple measurable quantity predicts the largest useful batch across tasks, and that it tends to grow as the loss falls, so the right batch changes over a run.
Boundary
Large runs where compute is billed by the accelerator-hour and a sweep is unaffordable.
Symptom or risk
Doubling blindly can leave the number of steps almost unchanged, or the batch you picked early becomes too large as the loss falls.
Exceptions
Estimates are noisy for small models; rely on the sweep in insight-1406 when runs are cheap.
Trade-offs
Needs per-example or per-microbatch gradient statistics, which complicate the training loop.
Pitfalls
Using one estimate from the first epoch for the whole run, since the value grows later.
Lessons and concepts
Claims and their sources
McCandlish et al. (arXiv preprint) report that the gradient noise scale predicts the largest batch size that still improves training and tends to rise as the loss falls.