Skip to content
ML Academy

Introduced on day 40

Optimizer state

The extra per-parameter tensors an optimizer keeps between steps, such as Adam's first and second moment estimates.

In detail

Adam stores two values per trainable parameter, so in fp32 the optimizer states are twice the size of the weights. They exist only for trainable parameters, which is why freezing layers cuts training memory more than lowering the working precision does.