Introduced on day 33
Gradient check
A test that compares analytic gradients with central-difference numerical gradients on a few examples, in double precision, to catch backward-pass bugs.
In detail
Perturb one parameter by plus and minus a small epsilon, take the loss difference over 2 epsilon, and compare with the analytic value using a relative error. It verifies the algebra of the code at one point, not the choice of model or loss, and it needs smooth points (ReLU kinks can fail it) and float64 arithmetic.