Neural networks, deep learning, NLP · insight-0437
Switch on deterministic GPU mode while bisecting a training regression
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- banking
Summary: situation, action, reason, boundary
Situation
After a refactor, validation accuracy of a credit-risk network drops by half a point, and you want to find the commit responsible.
Action
For the investigation only, fix all seeds, call
torch.use_deterministic_algorithms(True), setcudnn.benchmark = False, pin library versions and hardware, and confirm two runs of the known-good commit match before bisecting. Return to fast settings for normal training.Reason
These settings remove most within-platform nondeterminism, which turns metric differences between commits into evidence; PyTorch still does not guarantee identical results across releases or platforms.
Boundary
Debugging regressions, reproducing incidents and verifying refactors that should not change behaviour.
Symptom or risk
With nondeterministic GPU kernels and cuDNN autotuning, two runs of the same commit differ, so a bisect cannot tell a real regression from run-to-run noise.
Exceptions
If the regression only appears with nondeterministic kernels, compare distributions over several seeds instead.
Trade-offs
Deterministic algorithms can be slower, and some operations raise an error because no deterministic version exists.
Pitfalls
Bisecting on a different GPU type or library version than the original runs.
Lessons and concepts
Claims and their sources
PyTorch documents that completely reproducible results are not guaranteed across releases, commits or platforms, nor between CPU and GPU, and that torch.use_deterministic_algorithms and disabling cudnn.benchmark limit nondeterminism.