Neural networks, deep learning, NLP · insight-0417
Exclude biases and normalization scales from weight decay
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You apply AdamW to a model with LayerNorm or BatchNorm by passing
model.parameters()with one weight-decay value.Action
Build two parameter groups: weights of linear, convolutional and embedding layers with decay, and biases plus normalization parameters with
weight_decay=0.Reason
Weight decay is meant to limit the size of the weights that create interactions. Normalization gains and biases set offsets and scales, and shrinking them does not reduce model complexity in the same way.
Boundary
Transformer and CNN training with AdamW or SGD plus decay.
Symptom or risk
Decay pulls normalization gains and biases towards zero, which constrains the scale the network needs rather than its complexity; the effect shows up as slightly worse or less stable results.
Exceptions
Small models where measured differences are negligible; keep whichever your validation results support.
Trade-offs
More optimizer code and one more convention to keep consistent across experiments.
Pitfalls
Matching parameters by name fragments that miss renamed layers; assert the group sizes.
Lessons and concepts
Claims and their sources
PyTorch optimizers accept an iterable of dicts that define parameter groups with their own options.