Neural networks, deep learning, NLP · insight-1404
Start residual blocks as the identity by zero-initialising their last scale
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A deep residual network trains with a very large batch, or its loss is erratic during the first hundred steps even after warmup.
Action
Set the scale parameter of the final normalisation layer in each residual branch to zero, keep the others at one, and compare the first 200 steps with the default initialisation.
Reason
With that scale at zero, each block's output is the identity shortcut at step zero, so the network starts as a shallow function and the residual branches switch on as they are learned.
Boundary
Residual CNNs with a normalisation scale in the branch (the setting Goyal et al. tested), mostly large-batch training; Transformers are an untested extrapolation.
Symptom or risk
Every block contributes noise at initialisation, so the signal has to pass through many randomly initialised branches before the network can fit anything, and large-batch runs fall behind the small-batch baseline.
Exceptions
Networks without a normalisation scale in the branch need another mechanism, such as a learned gate.
Trade-offs
Weights below the zeroed scale receive no gradient at step zero, until the scale itself moves away from zero.
Pitfalls
Applying it to the shortcut path or to every normalisation layer disconnects the network.
Lessons and concepts
Claims and their sources
Goyal et al. initialise the scale of each residual block's last batch normalisation to zero so the signal initially propagates through the identity shortcut, and report that this improves optimisation, especially for large minibatches.