Skip to content
ML Academy

Neural networks, deep learning, NLP · insight-0430

Unscale gradients before clipping when training with a gradient scaler

Evidence type
documented fact
Difficulty
medium
Roles
ML engineer
Industries
cross-industry

Summary: situation, action, reason, boundary

  1. Situation

    You add gradient clipping to a training loop that already uses GradScaler for mixed precision.

  2. Action

    Call scaler.unscale_(optimizer) after backward and before clip_grad_norm_, then scaler.step and scaler.update. Log the unscaled norm, not the scaled one.

  3. Reason

    The PyTorch AMP examples state that clipping without unscaling uses scaled gradients and invalidates the threshold.

  4. Boundary

    Any loop combining a gradient scaler with clipping or with gradient-norm logging.

Symptom or risk

Clipping runs on scaled gradients, so the threshold means something different on every step as the scale changes; clipping either never fires or fires constantly.

Exceptions

Loops without a scaler; reduced-precision formats that you run without scaling.

Trade-offs

None beyond one extra call; it must happen once per optimizer per step.

Pitfalls

Calling unscale twice in one step, or unscaling only one of several optimizers.

Lessons and concepts

day 34day 37
Training dynamics and regularization

Claims and their sources