Neural networks, deep learning, NLP · insight-1420
Write a run manifest that lets a stranger repeat the experiment
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A model that scored well three weeks ago cannot be rebuilt: the notebook, the library versions and the data extract have all changed.
Action
For each run store the code revision, resolved dependency versions, the data snapshot identifier or hash, the full config, every seed, the hardware, and the metric files, and make the training script refuse to start without them.
Reason
A large conference's reproducibility programme combined code submission, a community reproduction challenge and a checklist. PyTorch also warns that results are not guaranteed identical across releases or platforms, so the environment has to be recorded to repeat a run.
Boundary
All experiments whose number will be cited, compared or deployed.
Symptom or risk
You cannot tell a real improvement from an environment change, and nobody can audit the number you quoted.
Exceptions
Throwaway exploration, which should be marked as such.
Trade-offs
Some friction per run; automate it in the launcher.
Pitfalls
Recording the config file name but not its resolved contents with defaults filled in.
Lessons and concepts
Claims and their sources
The NeurIPS 2019 reproducibility programme combined code submission, a community reproducibility challenge and a reproducibility checklist.
PyTorch states that completely reproducible results are not guaranteed across releases, commits or platforms.