Skip to content
ML Academy

Introduced on day 34

Weight decay

Shrinking weights toward zero by a small factor at each step; equal to an L2 penalty for plain SGD but different for adaptive optimizers.

In detail

With Adam an L2 term inside the gradient is divided by the adaptive denominator, so decay becomes uneven across parameters; AdamW applies the shrinkage directly, decoupled from the gradient step. Biases and normalization scales are commonly excluded. Too much decay underfits.