Introduced on day 33
Backpropagation
The algorithm that computes the gradient of the loss with respect to every parameter in one backward pass, by applying the chain rule through the computational graph.
In detail
It produces gradients only; the optimizer decides how to use them. Each parameter's gradient has the shape of the parameter. Its cost is a small multiple of a forward pass, and it needs the forward intermediates, so training memory grows with batch size and depth.