Neural networks, deep learning, NLP · insight-1437
Synchronise BatchNorm statistics when the per-device batch becomes small
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You scale a job from one GPU with batch 64 to eight GPUs with 8 examples each, using data-parallel training, and keep BatchNorm layers.
Action
Convert BatchNorm layers to synchronised BatchNorm before wrapping in DistributedDataParallel, or switch to a normalisation that is independent of the batch, and compare with the single-device baseline.
Reason
Standard BatchNorm in data-parallel training sees only the local mini-batch; synchronised BatchNorm computes statistics over the mini-batches of the whole process group, at the cost of communication.
Boundary
Multi-GPU training of convolutional networks with small per-device batches.
Symptom or risk
Each process computes statistics from its own few examples, so they are noisy, validation metrics drop, and the numbers differ from the single-device run you wanted to match.
Exceptions
Layer-normalised Transformers and models with large per-device batches.
Trade-offs
Synchronisation adds communication each step and supports one GPU per process only.
Pitfalls
Assuming synchronisation also applies at evaluation; it applies in training mode only.
Lessons and concepts
Claims and their sources
PyTorch SyncBatchNorm computes mean and standard deviation over the mini-batches of the process group, works with DistributedDataParallel with one GPU per process, and synchronises only in training mode.