Neural networks, deep learning, NLP · insight-0420

Make sure BatchNorm uses running statistics when you evaluate and serve

Situation
Validation accuracy differs between two scoring scripts, or a model scores the same image differently in a batch of one and in a full batch.
Action
Call model.eval() in every evaluation and serving path and add a test that one input scores identically alone and inside a batch.
Reason
BatchNorm keeps running estimates of mean and variance during training and uses them only in evaluation mode; eval() switches that mode for BatchNorm and Dropout.
Boundary
Any network with BatchNorm, including pretrained CNN backbones used for features.

Symptom or risk

The model is still in training mode, so BatchNorm normalizes with the current batch's statistics; predictions depend on what else is in the batch.

Test-time adaptation methods that deliberately recompute statistics; document them explicitly.

None for correctness; running statistics can be stale after fine-tuning, so recheck them after domain shifts.

Calling eval() before validation but forgetting train() afterwards, so later epochs train without dropout.

PyTorch BatchNorm keeps running estimates of mean and variance during training and uses them for normalization during evaluation.
BatchNorm2d (PyTorch documentation)
Module.eval() sets evaluation mode, which affects modules such as Dropout and BatchNorm.
Module (PyTorch documentation)