Neural networks, deep learning, NLP · insight-0419
Replace BatchNorm when memory forces very small batches
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- manufacturing, healthcare
Summary: situation, action, reason, boundary
Situation
High-resolution inspection images or long sequences force a batch size of 2 to 8 per GPU in a CNN that uses BatchNorm.
Action
Switch to GroupNorm for CNNs or LayerNorm for sequence models, whose statistics do not depend on the batch; or freeze BatchNorm statistics from a pretrained model.
Reason
Wu and He report BatchNorm error rising quickly as batch size shrinks, while GroupNorm stays stable across batch sizes. LayerNorm normalizes per example.
Boundary
Detection, segmentation, medical or industrial imaging and video, where memory limits batch size.
Symptom or risk
Batch statistics become noisy, training is unstable, and running averages used at inference no longer match the statistics seen during training.
Exceptions
With normal batch sizes, BatchNorm remains competitive and is the default in many pretrained CNNs.
Trade-offs
Swapping normalization invalidates pretrained BatchNorm weights; retraining or careful fine-tuning is needed.
Pitfalls
Using gradient accumulation and assuming it fixes BatchNorm; it enlarges the gradient batch, not the statistics batch.
Lessons and concepts
Claims and their sources
Wu and He report BatchNorm error increasing rapidly with smaller batches, and GroupNorm with 10.6% lower error than BatchNorm at batch size 2 on ResNet-50 ImageNet.
Layer normalization computes statistics per example, independent of batch size.