Introduced on day 33

Momentum

An optimizer modification that keeps a running velocity of past gradients and steps along it, speeding up consistent directions and damping oscillation.

In detail

With $v \leftarrow \beta v + g$ and $\theta \leftarrow \theta - \eta v$ , the first step equals plain SGD and later steps carry the past. Common values of $\beta$ are around 0.9. The gradient itself is unchanged; only the update rule differs.