Introduced on day 39
Layer normalization
Normalising each token's feature vector to zero mean and unit variance (with learned scale and shift), independent of the batch.
In detail
Unlike batch normalization it uses no batch statistics, so it behaves the same in training and inference and works with small batches and variable-length sequences. Transformers apply it around attention and feed-forward sublayers, either after the residual addition (post-LN) or before the sublayer (pre-LN).