Introduced on day 34
Weight initialization
The random scale at which weights start, chosen so that activations and gradients keep a stable size through the layers at step 0.
In detail
Glorot (Xavier) initialization balances variance across layers for tanh-like units; He initialization uses twice that variance for ReLU because half the inputs are zeroed. Too small a scale makes signals vanish and too large makes them explode, so a deep network can fail to train at all. Biases usually start at zero.