Neural networks, deep learning, NLP · insight-0421
Turn dropout off at inference unless you are deliberately sampling
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- ML engineer, Data scientist
- Industries
- retail, logistics
Summary: situation, action, reason, boundary
Situation
A deployed demand model returns slightly different predictions for the same request, and nobody changed the code.
Action
Put the model in evaluation mode for serving and evaluation, and add a test that two forward passes on the same input return identical outputs.
Reason
Dropout zeroes units with probability p and scales the rest by 1/(1-p) during training, so in evaluation mode it is an identity and no rescaling is needed.
Boundary
Any network with dropout layers, including Transformer fine-tunes.
Symptom or risk
Dropout is still active, so each call randomly zeroes units; predictions are noisy, reported accuracy is lower than possible and audits cannot reproduce a decision.
Exceptions
Monte Carlo dropout for uncertainty keeps dropout on by design and averages many passes.
Trade-offs
None; the test is cheap and protects reproducibility.
Pitfalls
Tracing a model (for example with torch.jit.trace) while it is still in training mode, which records active dropout in the graph.
Lessons and concepts
Claims and their sources
PyTorch Dropout scales outputs by 1/(1-p) during training, so in evaluation it computes an identity function.
Dropout was introduced as a regularizer with different behaviour at training and test time.