Neural networks, deep learning, NLP · insight-1427
Choose bfloat16 or float16 by range, not by speed alone, and remember float16 needs a scaler
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You enable mixed precision on a model that was pretrained or developed in bfloat16, or you move a working float32 job to float16 for speed.
Action
Use bfloat16 where hardware supports it, otherwise float16 with a gradient scaler. Log the scaler value and the count of skipped steps, and keep one float32 run as the reference for loss and metric.
Reason
Float16 gradients can underflow, so the loss is scaled before backward and gradients unscaled before the step; PyTorch notes that models pretrained in bfloat16 often cannot operate within the float16 range of 65,504.
Boundary
GPU training and inference with autocast in PyTorch.
Symptom or risk
Activations or logits exceed the float16 range, producing infinities and NaNs, or small gradients underflow to zero and layers silently stop learning.
Exceptions
Small models where float32 is already fast enough.
Trade-offs
bfloat16 keeps range but has fewer mantissa bits, so fine numerical detail is coarser.
Pitfalls
A scaler that keeps shrinking toward zero is a symptom of overflow, not a harmless warning.
Lessons and concepts
Claims and their sources
PyTorch documents that float16 gradients can underflow so gradient scaling multiplies the loss, and that most bfloat16-pretrained models cannot operate in float16's maximum range of 65504.