Introduced on day 34
Batch normalization
A layer that standardizes each feature over the current mini-batch and then applies a learned scale and shift, using running averages at inference.
In detail
It often allows larger learning rates and faster training. Its output for a row depends on the other rows in the batch during training, so evaluation mode must use the stored running statistics, and very small batches give noisy estimates. The original paper's explanation (internal covariate shift) is the authors' motivation, not a settled mechanism.