Neural networks, deep learning, NLP · insight-0428

Triage a NaN loss in a fixed order, starting with the data

Situation
The loss of a fraud model becomes NaN in epoch 3, and the team starts changing the architecture.
Action
Check in order: non-finite values in inputs and targets of the failing batch; operations such as log, division or sqrt on values near zero; learning rate and gradient norm; reduced-precision overflow. Use torch.autograd.detect_anomaly to locate the forward operation behind the failing backward.
Reason
Data and numerical edge cases cause most NaNs and are cheap to check; anomaly detection points to the exact operation instead of guessing.
Boundary
Any training run producing inf or NaN.

Symptom or risk

Changing several things at once hides the cause; a corrupt batch or a log of zero returns in the next retrain.

Divergence that grows smoothly before NaN is usually an optimization issue; start with the learning rate.

Anomaly detection slows training noticeably; enable it only for reproduction runs.

Replacing NaNs with zeros in the loss, which silently trains on broken batches.

torch.autograd.detect_anomaly makes the backward pass report the traceback of the forward operation that created the failing function and, with check_nan, raises when backward produces NaN.
Automatic differentiation package - torch.autograd