Neural networks, deep learning, NLP · insight-0427

Diagnose vanishing gradients with per-layer gradient norms, then fix the architecture

Situation
A deep plain network trains no better than a shallower one, or its early layers barely change from initialization.
Action
Plot gradient norm per layer for the first few hundred steps. If early layers are orders of magnitude smaller, add residual connections, normalization, non-saturating activations or gated units before adding more depth.
Reason
Repeated multiplication through many layers or time steps shrinks gradients; residual paths give gradients a shorter route, which helps very deep residual networks train.
Boundary
Deep MLPs and CNNs, and RNNs over long sequences.

Symptom or risk

Gradients shrink as they flow backward, so early layers learn little; adding depth then adds cost without benefit.

Small differences between layers are normal; act on orders of magnitude, not on ratios of two or three.

Residual and normalization layers add parameters and code paths that complicate exporting.

Raising the learning rate to compensate, which explodes the later layers instead.

Bengio, Simard and Frasconi showed that learning long-term dependencies with gradient descent is difficult because gradients vanish or explode.
Learning long-term dependencies with gradient descent is difficult
He et al. showed residual connections make very deep networks trainable.
Deep Residual Learning for Image Recognition