Neural networks, deep learning, NLP · insight-1408
Do not place dropout directly before a BatchNorm layer
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A convolutional or dense network has a dropout layer immediately followed by BatchNorm and validation accuracy is slightly worse in evaluation mode than the training curve suggests.
Action
Move dropout after the last normalisation layer, or remove it where BatchNorm already regularises. Compare evaluation-mode validation loss for both orderings over three seeds.
Reason
Li and colleagues explain the degradation with a variance shift between the training and test phases and test modified dropout schemes that reduce it.
Boundary
Convolutional and dense networks using BatchNorm with standard dropout.
Symptom or risk
Dropout changes the variance a unit sees between training and inference, while BatchNorm keeps the variance measured in training, so inference-time statistics are mismatched and errors rise.
Exceptions
Layer normalisation and group normalisation use per-example statistics, so the mismatch does not arise in the same way.
Trade-offs
Moving dropout later changes which activations are regularised.
Pitfalls
Treating a small gap as proof; the effect is architecture-dependent and needs the seed comparison.
Lessons and concepts
Claims and their sources
Li et al. (arXiv preprint) attribute the disharmony between dropout and batch normalisation to a variance shift between training and test, and evaluate modified dropout strategies.