Skip to content
ML Academy

Neural networks, deep learning, NLP · insight-1405

Keep a frozen backbone's BatchNorm layers in inference mode while you fine-tune

Evidence type
documented fact
Difficulty
medium
Roles
ML engineer, Applied AI
Industries
cross-industry

Summary: situation, action, reason, boundary

  1. Situation

    You freeze a pretrained image backbone, train a new head on a few thousand labelled images, then unfreeze the top blocks.

  2. Action

    Call the backbone in inference mode (Keras training=False; in PyTorch put those layers in eval()) and check the running statistics are identical before and after one epoch.

  3. Reason

    The Keras guide documents that BatchNorm statistics update during training and that the base model should be called in inference mode during fine-tuning so the learned statistics are not disturbed.

  4. Boundary

    Fine-tuning any normalised pretrained network with batches much smaller or less diverse than pretraining.

Symptom or risk

Freezing weights does not always freeze the running mean and variance, so batch statistics of your small, different dataset overwrite the pretrained ones and accuracy drops the moment you unfreeze.

Exceptions

Large, representative target data may justify updating statistics, even with BatchNorm unfrozen on purpose.

Trade-offs

Frozen statistics adapt slowly to a genuinely different input distribution.

Pitfalls

A later model.train() call re-enables updates for every submodule; re-apply the inference setting after it.

Lessons and concepts

day 40
Transfer learning and fine-tuning decisions

Claims and their sources

  • The Keras transfer-learning guide says BatchNormalization keeps non-trainable running statistics and that a base model being fine-tuned should be called with training=False so those statistics are not updated.

    Transfer learning and fine-tuning (Keras guide)