Neural networks, deep learning, NLP · insight-1428
Keep a float32 master copy of the weights, and compute losses and reductions in float32
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You cast an entire model and optimizer to half precision to fit a bigger batch, and the loss stops improving after the learning rate decays.
Action
Use autocast with float32 parameters, or keep an explicit float32 master weight copy that receives the update. Check the metric against a float32 run on a short schedule.
Reason
Mixed-precision training keeps a float32 master copy of the weights and scales the loss to handle half precision's range limits; PyTorch's autocast runs losses, softmax and reductions in float32.
Boundary
Half-precision training of any network, and custom training loops without a framework wrapper.
Symptom or risk
Small updates are rounded away when added to larger half-precision weights, so late-stage learning stalls; softmax, loss sums and norms lose accuracy.
Exceptions
Pure inference after training, where half precision weights are fine once validated.
Trade-offs
Master weights add memory for a float32 copy.
Pitfalls
Calling .half() on the whole model, which disables the safeguards autocast provides.
Lessons and concepts
Claims and their sources
Micikevicius et al. propose keeping a single-precision master copy of the weights and loss scaling to handle half-precision range limits.
PyTorch autocast runs softmax, normalisation, several losses and reductions in float32.