Neural networks, deep learning, NLP · insight-0426
Log the global gradient norm and clip it when spikes precede loss blow-ups
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Training loss on a recurrent or deep model occasionally jumps by orders of magnitude, sometimes ending in NaN.
Action
Log the total gradient norm every step. If spikes line up with the loss jumps, apply
clip_grad_norm_with a threshold slightly above the typical norm, and keep logging how often clipping fires.Reason
Pascanu et al. proposed gradient-norm clipping for exploding gradients in recurrent networks; it rescales only the rare large updates and leaves normal steps untouched.
Boundary
RNNs, LSTMs, Transformers and any model with occasional loss spikes.
Symptom or risk
Rare exploding gradients produce one huge update that destroys learned weights; lowering the learning rate for the whole run fixes it at the cost of slow training.
Exceptions
If clipping fires on most steps, the threshold is too low or the learning rate too high; fix those instead.
Trade-offs
Clipping changes the effective step size and can hide data problems such as corrupt batches.
Pitfalls
Clipping each parameter separately when you meant one global norm; not logging clip frequency.
Lessons and concepts
Claims and their sources
Pascanu, Mikolov and Bengio propose a gradient norm clipping strategy to deal with exploding gradients.
PyTorch clip_grad_norm_ clips the norm computed over all parameters' gradients as if concatenated into one vector, modifying gradients in place.