Neural networks, deep learning, NLP · insight-0429

Use a gradient scaler with float16 mixed precision and compare against a float32 run

Situation
You enable float16 autocast to cut memory and speed up training of an image or text model.
Action
Use torch.amp autocast together with a gradient scaler, keep the optimizer's master weights in float32, and run one short float32 reference to confirm validation metrics match within noise.
Reason
Micikevicius et al. made half-precision training work with a float32 master copy of weights and loss scaling; the PyTorch AMP docs describe gradient scaling as the remedy for float16 gradient underflow.
Boundary
GPU training with float16 autocast.

Symptom or risk

Small gradients underflow to zero in float16, so some parameters stop updating; the run looks fine but ends worse than float32, and nobody compares.

Pure inference does not need gradient scaling; check accuracy on validation data instead.

Memory and speed gains depend on hardware; a reference run costs time once.

Casting the whole model with .half() instead of using autocast, which also puts the loss and reductions in float16.

Micikevicius et al. train in half precision by keeping a single-precision master copy of weights and scaling the loss.
Mixed Precision Training
PyTorch documents that small float16 gradients can flush to zero and that gradient scaling multiplies the loss to prevent underflow.
Automatic Mixed Precision package - torch.amp (PyTorch documentation)