Introduced on day 37

Vanishing gradient through time

The gradient from early steps of a sequence is multiplied by a factor below 1 at every step, so it shrinks geometrically and early steps get almost no learning signal.

In detail

In a vanilla recurrent cell the gradient of $h_T$ with respect to $h_1$ is a product of $T-1$ per-step terms. Below 1 the signal vanishes; above 1 it explodes, and clipping the gradient norm is the usual remedy. Gated cells (LSTM, GRU), shorter paths (attention) and truncation reduce the problem without removing the need to test long dependencies.