Introduced on day 32
Saturation
The flat tails of an activation such as sigmoid or tanh, where the output barely moves with the input and the gradient passing through is close to zero.
In detail
A saturated unit learns slowly because its local slope is near 0, and a deep stack of saturating units shrinks the backward signal geometrically. Large inputs, large weights and poor initialization all push units into saturation; ReLU does not saturate for positive inputs but has an exactly flat negative side.