Neural networks, deep learning, NLP · insight-0416
With Adam, use decoupled weight decay (AdamW), not the weight_decay L2 term
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You add regularization to an Adam-trained model by setting
weight_decayintorch.optim.Adam.Action
Switch to
torch.optim.AdamW(orAdam(decoupled_weight_decay=True)in recent PyTorch releases), retune the decay coefficient, and note that AdamW's default decay is 0.01, not 0.Reason
Loshchilov and Hutter showed L2 regularization and weight decay are equivalent for SGD but not for adaptive optimizers; decoupling applies the decay directly to the weights.
Boundary
Any Adam-family training with regularization, including Transformer fine-tuning.
Symptom or risk
In Adam the L2 gradient is rescaled by the adaptive denominator, so parameters with large historical gradients are barely regularized and tuning weight_decay behaves unpredictably.
Exceptions
Plain SGD, where L2 and weight decay coincide up to learning-rate scaling.
Trade-offs
Old decay values do not transfer; budget a small sweep.
Pitfalls
Switching to AdamW and silently inheriting a decay of 0.01 on parameters that should not be decayed.
Lessons and concepts
Claims and their sources
Loshchilov and Hutter show L2 regularization and weight decay differ for adaptive optimizers and propose decoupled weight decay (AdamW).
PyTorch documents Adam's weight_decay as an L2 penalty (default 0) and AdamW's weight_decay coefficient with default 1e-2.