Skip to content
ML Academy

Introduced on day 40

Mixed-precision training

Training with half-precision (fp16 or bf16) working tensors while keeping sensitive values such as master weights and reductions in float32.

In detail

Savings come mainly from activations and faster arithmetic. For Adam the per-parameter state is still about 16 bytes (2 + 2 working weights and gradients, 4 + 4 + 4 master weights and moments). fp16 needs loss scaling because small gradients underflow; bf16 has float32's range and usually does not. Always compare validation against a float32 reference run.