Skip to content
ML Academy

Introduced on day 33

Adam

An optimizer that scales each parameter's step by running averages of its gradient and squared gradient, with bias correction for the early steps.

In detail

The update is η m^/(v^+ϵ)\eta\, \hat m / (\sqrt{\hat v} + \epsilon). Because of the normalization, early steps are about the learning rate in size for every parameter, whatever the gradient scale. It is forgiving to learning-rate and scale differences, uses two extra values per parameter, and with weight decay should be used in its decoupled form (AdamW).