Neural networks, deep learning, NLP · insight-0425
Gradient-check custom backward code in double precision
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You implement a custom autograd function or a hand-written backward pass, for example for a custom pricing-aware loss.
Action
Run
torch.autograd.gradcheckon small double-precision inputs, away from kinks such as zero for ReLU or ties in max, and keep it as a unit test.Reason
Gradcheck compares analytical gradients with finite differences; its default tolerances assume float64 and it may fail at non-differentiable points.
Boundary
Custom autograd functions, custom CUDA kernels and manual gradient derivations.
Symptom or risk
A wrong gradient still lets loss fall a little, so the model trains to a worse solution and nobody suspects the math.
Exceptions
Standard layers and losses composed from built-in operations are already tested by the framework.
Trade-offs
Finite differences are slow, so test on tiny tensors.
Pitfalls
Running in float32 and loosening tolerances until the test passes.
Lessons and concepts
Claims and their sources
PyTorch gradcheck default tolerances are designed for double precision, the check will likely fail with lower precision, and it may fail at non-differentiable points.