Neural networks, deep learning, NLP · insight-0411
Match weight initialization to the activation function
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A deep MLP or CNN with ReLU activations trains very slowly, or a tanh network's loss does not move for many steps.
Action
Use He (Kaiming) initialization for ReLU-family layers and Glorot (Xavier) for tanh or sigmoid layers. Log the standard deviation of each layer's activations on one batch at initialization.
Reason
Both schemes choose weight variance so that signal variance is roughly preserved through layers for their activation type.
Boundary
Networks you build from scratch, especially deeper than a few layers.
Symptom or risk
Activation variance shrinks or grows with depth, so deep layers receive vanishing or exploding signals before any learning happens.
Exceptions
Pretrained weights replace initialization; normalization layers and residual connections reduce sensitivity.
Trade-offs
Framework defaults may already fit your activation; check before overriding.
Pitfalls
Changing the activation from tanh to ReLU and keeping the old initialization.
Lessons and concepts
Claims and their sources
Glorot and Bengio analyzed saturating activations and proposed variance-preserving initialization.
He et al. derived an initialization for ReLU-family networks.