Skip to content
ML Academy

Core day 33 of 60 · Week 7 · Part II · application depth

Backpropagation and Optimization

Your Day 31 network, now ending in a sigmoid, gives a positive example (label 1) a probability of 0.5. Which of its 13 numbers should change, in which direction and by how much, and when you switch from SGD to Adam does the gradient change, or only what you do with it?

By the end: For a small network and one example you compute every gradient with the chain rule, confirm it against a numerical gradient, apply one SGD update, and say how momentum and Adam would use the same gradient differently.

At a glance

Backpropagation multiplies local slopes backwards to get every gradient at once; the optimizer only decides what step to take with them.

  • Chain rule

    Each box multiplies the incoming gradient by its own local slope.

  • Shapes check

    Every gradient has the shape of its parameter; a mismatch is a bug.

  • Gradient check

    Compare with central differences in float64; here they agree to about 1e-10.

  • Update rules

    SGD steps by the gradient; momentum adds velocity; Adam rescales per parameter.

  • Through depth

    Products of slopes below 1 vanish, above 1 explode: check per-layer norms.

Recall · 5 min

Reviews due from Days 11, 31 and 32 appear first. If none are due: state the gradient of binary cross-entropy with respect to the logit, and the shape of ∂L/∂W2\partial L / \partial W_2 for the 2-3-1 network (W2W_2 has shape (3, 1)). Then differentiate f(z)=(3z+1)2f(z) = (3z + 1)^2 with the chain rule and check the result with a finite difference.

Open due reviews

Sign in to record this block.

Concept · 10 min

Training needs one number per parameter: nudge this weight up, how does the loss change? Nudging each weight and re-running the network costs one forward pass per parameter. Backpropagation gets every gradient from one forward and one backward pass. Then a separate job: the optimizer decides the step. The optimizer changes the update, not the gradient.

Backpropagation fills in one gradient per parameter; the optimizer then decides the step

Predict first

At the first step, SGD with momentum 0.9 and plain SGD start from zero velocity with the same gradient g. How do their first updates compare?

Text description of the figure

A step-through of one example in a 2-2-1 network (two hidden units, not the Day 31 network). It shows the forward values, then the local gradient at each box going backwards, the resulting gradient of every weight checked against a numerical gradient, and the parameter update under a chosen optimizer and learning rate, with the new loss after the step.

One training step, as a loop: only the last stop differs between optimizers Loop
1Forward passstore every Z and H forlater2Lossone number for the batch3Backward passone gradient perparameter, same shape4Optimizer stepSGD, momentum or Adamscoredifferentiatehand over gnext batch
  1. Forward passstore every Z and H for laterscore ↓
  2. Lossone number for the batchdifferentiate ↓
  3. Backward passone gradient per parameter, same shapehand over g ↓
  4. Optimizer stepSGD, momentum or Adamnext batch ↓
  5. ↺ back to 1 · Forward pass

The backward pass reuses the stored forward values and returns the same g whatever optimizer follows. Changing the optimizer changes only the fourth stop.

Forward pass (top) and backward pass (below) of the 2-3-1 network Diagram
x = (1, 2)input rowZ1 = xW1 + b1(2, 1.5, -1.4)H = ReLU(Z1)(2, 1.5, 0)p = sigmoid(HW2 +b2)logit 0, so p = 0.5Loss-log p = 0.6931dL/dzp - y = -0.5dL/dZ1via W2^T (-0.5, 0.5,-1), ReLU slopes (1, 1,0)dL/dW2H^T (p - y) = (-1,-0.75, 0)dL/dW1x^T dL/dZ1: (-0.5,0.5, 0), (-1, 1,0)backwardtimes H^Ttimes x^T
  1. x = (1, 2)input row
  2. Z1 = xW1 + b1(2, 1.5, -1.4)
  3. H = ReLU(Z1)(2, 1.5, 0)
    • to p = sigmoid(HW2 + b2)
  4. dL/dZ1via W2^T (-0.5, 0.5, -1), ReLU slopes (1, 1, 0)
    times x^T
  5. dL/dW1x^T dL/dZ1: (-0.5, 0.5, 0), (-1, 1, 0)
  6. p = sigmoid(HW2 + b2)logit 0, so p = 0.5
    • to Loss
  7. dL/dzp - y = -0.5
    • back to dL/dZ1
    times H^T
  8. dL/dW2H^T (p - y) = (-1, -0.75, 0)
  9. Loss-log p = 0.6931
    • backward: back to dL/dz

The backward pass starts at the loss and reuses every value stored on the way forward. dL/dZ1 is the output gradient sent back through W2^T, then multiplied by the ReLU slopes; highlighted boxes are the parameter gradients.

The backward pass in five multiplications Sequence

  1. Output

    Sigmoid plus cross-entropy: dL/dz = p - y = 0.5 - 1 = -0.5

  2. W2 and b2

    dW2 = H^T (p - y) = (2, 1.5, 0) x (-0.5) = (-1, -0.75, 0); db2 = -0.5

  3. Into the hidden layer

    dL/dH = -0.5 x (1, -1, 2) = (-0.5, 0.5, -1)

  4. Through ReLU

    Multiply by slopes (1, 1, 0): unit 3 was off, so it passes nothing

  5. W1 and b1

    dW1 = x^T dZ1: rows (-0.5, 0.5, 0) and (-1, 1, 0); db1 = (-0.5, 0.5, 0)

These are exactly the analytic gradients the application block prints and checks numerically.

Same gradient, three update rules Comparison
CriterionSGDMomentumAdam
Updatetheta minus lr times gv = beta v + g, then theta minus lr times vtheta minus lr times m-hat / (sqrt(v-hat) + eps)
Extra state per parameterNone1 (velocity)2 (first and second moments)
First step on W2 here (lr 0.1)(0.1, 0.075, 0)(0.1, 0.075, 0): identical at step 1(0.1, 0.1, 0): about lr times sign(g)
Typical strengthSimple; with a tuned schedule often generalizes wellSpeeds up consistent directions, damps oscillationRobust to gradient scale; a common default
Watch out forSlow along shallow directions, zig-zags in narrow valleysOvershoot when the rate is too highMemory for the moments; can generalize worse on some tasks

The gradient row is identical in all three columns. Only what happens after it differs.

Same gradients, three update rules, two landscapes Illustration
Narrow valley: f = ½(x² + 50y²)start

50 steps from the same start. SGD (rate 0.038) ends at f = 0.167; Momentum 0.9 (rate 0.004) ends at f = 0.051; Adam (rate 0.2) ends at f = 0.029.

Rosenbrock: f = (1 − x)² + 100(y − x²)²start

1500 steps from the same start. SGD (rate 0.0015) ends at f = 0.056; Momentum 0.9 (rate 0.0003) ends at f = 0.005; Adam (rate 0.02) ends at f = 0.012.

Rosenbrock loss, log10 scale1e-61e-41e-21e01e2050010001500step

The same three runs as loss curves. Each rule was given its own rate: one shared rate would make one of them crawl or blow up.

  • SGD
  • Momentum 0.9
  • Adam
  • contours of equal loss; + marks the minimum

Computed: exact gradients of each surface, one update rule per colour (momentum in the PyTorch form v = 0.9v + g; Adam with β = 0.9, 0.999). Rates were picked by hand for a readable picture, not tuned; a different rate can reorder the three.

SGD zig-zags across a narrow valley and crawls along its floor; momentum builds speed along the floor; Adam rescales each coordinate. On the banana-shaped Rosenbrock valley the ranking depends on the rates.

What ten layers do to the signal, forward and backward Illustration
Activation spread (std) after each layer1e-11e01e112345678910layer (1 = nearest the input)

Forward: tanh with Glorot drifts from 0.61 to 0.25; ReLU with He stays between 0.65 and 1.44; sigmoid outputs sit near 0.5 with a small spread.

Gradient norm of each weight matrix1e-51e-41e-31e-21e-11e01e11e212345678910layer (1 = nearest the input)

Backward, layer 1 versus layer 10: sigmoid 1/7,426, tanh 0.68, ReLU with He 1.28. Each sigmoid layer multiplies the signal by a slope of at most 0.25.

  • Sigmoid, weights N(0, 0.5)
  • Tanh, Glorot sd √(1/20)
  • ReLU, He sd √(2/20)

Computed: one seeded draw of weights and a batch of 64 standard-normal inputs, no biases, output gradient (a − 1)/64 as in the hands-on lab. Exact values change with the seed; the slopes of the lines do not.

The backward signal is multiplied by weights and slopes at every layer. Sigmoid slopes of at most 0.25 compound into a vanishing gradient; ReLU with He initialization keeps every layer in range.

  • Read the network as a Computational graph; going backwards, each box multiplies the incoming gradient by its own local derivative (the chain rule, Rumelhart, Hinton and Williams, 1986).
  • Every gradient has the shape of its parameter: a free check.
  • Update rules with learning rate η\eta: SGD, Momentum and Adam (Kingma and Ba, 2015), compared in the pictures.
  • Through depth, the backward signal is multiplied by weights and slopes at every layer; sigmoid slopes are at most 0.25, so deep stacks shrink it (Vanishing gradient) or large weights blow it up.

Assumptions

  • The loss is differentiable almost everywhere; ReLU's kink at 0 is handled by convention, and argmax or sampling block gradients.

  • A mini-batch gradient is a noisy estimate of the full-data gradient, which is one reason learning rate and batch size interact.

  • A Gradient check verifies the algebra at one point in double precision; it says nothing about whether the model or loss is right.

  • Optimizer comparisons mean something only when each optimizer's learning rate is tuned.

Sign in to record this block.

Application and workplace decision · 10 min

Work the Day 31 network one step further. Row 0, x=(1,2)x = (1, 2), label 1, logit 0, so p=0.5p = 0.5. On paper, predict:

  • the sign of ∂L/∂z\partial L / \partial z;
  • the three entries of ∂L/∂W2\partial L / \partial W_2, given hidden activations (2,1.5,0)(2, 1.5, 0);
  • which hidden unit passes nothing back to W1W_1.

The block then runs the backward pass, a central-difference check of every gradient, one SGD step with η=0.1\eta = 0.1, and the first update SGD and Adam would make to W2W_2.

Python

import numpy as np
# Day 31 network (2-3-1) with a sigmoid output; one booking x labelled no-show (y = 1)
P = {"W1": np.array([[1.0, -1.0, 0.5], [0.5, 1.0, -1.0]]), "b1": np.array([0.0, 0.5, 0.1]),
     "W2": np.array([[1.0], [-1.0], [2.0]]), "b2": np.array([-0.5])}
x, y = np.array([[1.0, 2.0]]), 1.0
def forward(P):
    Z1 = x @ P["W1"] + P["b1"]
    H = np.maximum(0, Z1)
    p = 1 / (1 + np.exp(-(H @ P["W2"] + P["b2"])))
    return Z1, H, p, float(-(y * np.log(p) + (1 - y) * np.log(1 - p))[0, 0])
Z1, H, p, loss = forward(P)
dz = p - y                                  # sigmoid + cross-entropy: dL/dlogit = p - y
dZ1 = (dz @ P["W2"].T) * (Z1 > 0)           # back through W2, then ReLU passes gradient where Z1 > 0
G = {"W2": H.T @ dz, "b2": dz.sum(0), "W1": x.T @ dZ1, "b1": dZ1.sum(0)}
def numeric(name, eps=1e-6):                # central difference, one parameter at a time
    g = np.zeros_like(P[name])
    for i in np.ndindex(g.shape):
        P[name][i] += eps; hi = forward(P)[3]
        P[name][i] -= 2 * eps; lo = forward(P)[3]
        P[name][i] += eps
        g[i] = (hi - lo) / (2 * eps)
    return g
print(f"forward: p = {p[0, 0]:.3f}, loss = {loss:.4f}")
for name in G:
    print(f"{name}: analytic {np.round(G[name].ravel(), 3)}, max |analytic - numeric| = {np.abs(G[name] - numeric(name)).max():.1e}")
lr = 0.1                                    # SGD: parameter minus learning rate times gradient
P2 = {k: P[k] - lr * G[k] for k in P}
print(f"after one SGD step (lr {lr}): loss {forward(P2)[3]:.4f}")
g = G["W2"].ravel()                         # same gradient, different update rule (first step)
print("SGD update ", np.round(-lr * g, 3) + 0.0)
print("Adam update", np.round(-lr * g / (np.abs(g) + 1e-8), 3) + 0.0)

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Sign in to record this block.

Mini-assessment · 8 min

Your first answer is your evidence. You can re-answer later in practice mode; that is labelled practice.

Question 1 · easy

Which statement correctly splits the roles in one training step?
Options

How confident are you? (optional, used for calibration)

Question 2 · medium

SGD with momentum uses $v \leftarrow \beta v + g$ and then $\theta \leftarrow \theta - \eta v$, with $\beta = 0.9$, $\eta = 0.1$ and $v = 0$ at the start. A parameter's gradients on the first two steps are 4 and 2. By how much does the parameter change on the second step (absolute value)?
parameter units

How confident are you? (optional, used for calibration)

Question 3 · hard

A 10-layer sigmoid network (width 20, weights drawn from a normal with standard deviation 0.5) for fare-class prediction does not train; the loss stays near its initial value. One backward pass prints the gradient norm of each weight matrix: layer 1 (input side) $9 \times 10^{-5}$, layer 5 $2 \times 10^{-2}$, layer 10 (output side) 1.1. What is the most likely diagnosis and the best first change?
Options

How confident are you? (optional, used for calibration)

Question 4 · medium

Take the Day 31 network with hand-set weights, a sigmoid output and cross-entropy loss. A booking with $x = (2, -1)$ is labelled $y = 0$ (the passenger showed up). The forward pass gives hidden activations $(1.5, 0, 2.1)$, logit 5.2 and $p = 0.9945$. What is the gradient of the loss with respect to $W_1[1, 2]$, the weight from the second input to the third hidden unit? (Two decimals, signed.)
gradient

How confident are you? (optional, used for calibration)

Question 5 · hard

A teammate hand-writes the backward pass for the Day 33 network and forgets one factor. On the booking $x = (1, 2)$ with $y = 1$, a double-precision gradient check prints the largest absolute difference between the analytic gradient and central differences per array: $W_2$ $6 \times 10^{-11}$, $b_2$ $4 \times 10^{-11}$, $W_1$ 2.0, $b_1$ 1.0. The mismatch sits only in the third column of $W_1$ and the third entry of $b_1$, the unit whose pre-activation is $-1.4$; there the analytic values are non-zero and the numerical ones are 0. What is the bug, and what is the next step?
Options

How confident are you? (optional, used for calibration)

Sign in to complete lessons and keep your progress.

Technical depth · optional · 26 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The chain rule, box by box4 minBackpropagation written as an algorithm for any depth, the single formula for the first-layer gradient, and the local rule of each box.
Backpropagation for any stack of dense layers, as an algorithm Sequence

  1. Forward, and keep

    For l = 1 to L: Z_l = H_(l-1) W_l + b_l and H_l = f(Z_l), starting from H_0 = X. Store every Z_l and H_l

  2. Seed the output

    Softmax or sigmoid with cross-entropy, averaged over a batch of B rows, gives D_L = (P - Y) / B (Day 32)

  3. Weight gradients

    dW_l = H_(l-1)^T D_l and db_l = D_l summed over the rows: exactly the shapes of W_l and b_l

  4. Pass the signal down

    D_(l-1) = (D_l W_l^T) times f'(Z_(l-1)) entry by entry: back through the weights, then through the activation's slope

  5. Repeat

    Step down to l - 1 until l = 1. Each layer is visited once, in reverse order, reusing what the forward pass stored

  6. Cost

    Two matrix products per layer (dW_l and D_(l-1)) against one forward: about twice the forward cost, plus every stored Z and H

Today's example is L = 2 and B = 1: D_2 = p - y = -0.5, dW_2 = H^T D_2, D_1 = (D_2 W_2^T) times 1[Z_1 > 0]. Autograd in every framework runs this loop for you.

The whole backward pass for one example Formula
∂L∂W1=x⊤[((p−y) W2⊤)⊙1[Z1>0]]\frac{\partial L}{\partial W_1} = x^\top \Big[ \big( (p - y)\, W_2^\top \big) \odot \mathbb{1}[Z_1 > 0] \Big]
p−yp - y
gradient at the logit (Day 32)
W2⊤W_2^\top
sends the output gradient back to each hidden unit
1[Z1>0]\mathbb{1}[Z_1 > 0]
ReLU slope: 1 if the unit was active, 0 if not
x⊤x^\top
turns the hidden-unit gradients into one gradient per weight
Each box only needs its own local rule Table
Box in the forward passBackward rule (G = incoming gradient)Shape check
Matrix product A = XWdX = G W^T and dW = X^T GdW has the shape of W
Bias A = Z + bdb = G summed over the batch rowsdb has the shape of b
ReLUG times 1 where the input was positive, else 0same as the input
Sigmoid then cross-entropyp - yone value per row
Shapes of the gradients2 minThe free check that catches most hand-written backward bugs.
Every gradient has the shape of its parameter Table
ParameterShapeGradient here
W1(2, 3)rows (-0.5, 0.5, 0) and (-1, 1, 0)
b1(3,)(-0.5, 0.5, 0)
W2(3, 1)(-1, -0.75, 0) as a column
b2(1,)(-0.5)
Why reverse mode, and what it costsadvanced3 minFinite differences, forward mode and reverse mode compared on cost, memory and exactness.
Three ways to get N gradients Comparison
CriterionFinite differencesForward-mode autodiffReverse mode (backprop)
Cost for one loss and N parameters2N forward passes (central)About N passes, one per input directionOne forward plus one backward, a small multiple of a forward pass
MemoryLowLowStores forward intermediates
Exact?No: truncation and rounding errorYes, to roundingYes, to rounding
Where you meet itGradient checks onlyFew inputs, many outputsTraining: one loss, millions of parameters

Reverse mode wins whenever there is one scalar loss and many parameters, at the price of keeping activations (Day 31's memory note).

Reverse mode buys speed with memory; checkpointing buys some of it back Scale
  1. Autograd defaultkeeps every activation until backward
  2. Checkpointingstores a few, recomputes the rest (Day 40)

Frameworks record the forward graph and replay it backwards (PyTorch autograd mechanics); that recording is why activations stay in memory. Checkpointing trades the other way: less memory for extra forward compute during backward.

Running a gradient check4 minCentral differences, the right epsilon and how to localize a failure.
The numerical gradient used in the check Formula
gi≈L(θ+ϵei)−L(θ−ϵei)2ϵ,error=O(ϵ2)g_i \approx \frac{L(\theta + \epsilon e_i) - L(\theta - \epsilon e_i)}{2\epsilon}, \quad \text{error} = O(\epsilon^2)
eie_i
a vector that is 1 for parameter i and 0 elsewhere
ϵ\epsilon
about 1e-6 in float64; much smaller steps let rounding error (about 1e-16 / epsilon) take over
O(ϵ2)O(\epsilon^2)
a one-sided difference only reaches O(epsilon)
Choosing the step of a gradient check Illustration
1e01e-21e-41e-61e-81e-101e-121e-11e-31e-51e-71e-91e-111e-13step ε (smaller to the right)largest relative error, 9 parameters← truncation: error ∝ ε (one-sided), ∝ ε² (central)rounding: error ∝ 1e-16 / ε →sweet spot
  • central difference (f(θ+ε) − f(θ−ε)) / 2ε
  • one-sided difference (f(θ+ε) − f(θ)) / ε

Computed in float64 on the backprop stepper's 2-2-1 sigmoid network (one example, its starting weights): largest relative error |a − n| / (|a| + |n|) over the nine parameters.

Central differences reach a relative error of 4e-11 at ε ≈ 2e-5; one-sided differences bottom out at 1e-8 near ε ≈ 1e-7. Left of the minimum the formula's own error dominates; right of it, rounding in float64 does.

Too large a step measures the curvature, too small a step measures rounding. Central differences tolerate a wider range and reach a far smaller error.

How to run a gradient check, and the two ways it lies Diagram
Few rows,float645 to 10 examplesAny |Z| withineps of 0?Pick other rowsat a kink the numericReLU slope blends 0 and1Analytic vsnumericalyour backward pass vscentral differencesMax error per arrayW1, b1, W2, b2Fix that backwardstepthe failing array namesthe layerTune the optimizeronly nowyesretrynofar above 1e-5about 1e-10
  1. Few rows, float645 to 10 examples
  2. Any |Z| within eps of 0?
    • no: to Analytic vs numerical
    yes
  3. Pick other rowsat a kink the numeric ReLU slope blends 0 and 1
    • retry: back to Few rows, float64
  4. Analytic vs numericalyour backward pass vs central differences
  5. Max error per arrayW1, b1, W2, b2
    • about 1e-10: to Tune the optimizer
    far above 1e-5
  6. Fix that backward stepthe failing array names the layer
  7. Tune the optimizeronly now

Check in float64: PyTorch's gradcheck documents that its default tolerances assume double precision. Report the largest error per parameter array, so a failure points at a layer. Today every array agrees to about 1e-10; the 1e-5 alarm level is a rule of thumb.

The update rules, each worked by hand4 minSGD, momentum and Adam as equations, two hand-computed steps of each on the same gradients, and the published defaults.
SGD and momentum, the update rules Formula
θ←θ−η g(SGD),v←βv+g,  θ←θ−η v(momentum)\theta \leftarrow \theta - \eta\, g \quad\text{(SGD)}, \qquad v \leftarrow \beta v + g,\; \theta \leftarrow \theta - \eta\, v \quad\text{(momentum)}
gg
the gradient backpropagation returned; identical for every rule
η\eta
learning rate: the step per unit of gradient (or of velocity)
vv
velocity, a running sum that decays by beta each step; starts at 0
β\beta
usually 0.9: a steady gradient builds v up to about g / (1 - beta) = 10 g

Momentum's first step equals SGD's, because the velocity starts at zero.

Adam's update Formula
mt=β1mt−1+(1−β1)g,  vt=β2vt−1+(1−β2)g2,  θ←θ−ηm^tv^t+ϵm_t = \beta_1 m_{t-1} + (1-\beta_1) g, \; v_t = \beta_2 v_{t-1} + (1-\beta_2) g^2, \; \theta \leftarrow \theta - \eta \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}
mt,vtm_t, v_t
running averages of the gradient and the squared gradient
m^t,v^t\hat m_t, \hat v_t
bias-corrected: divided by 1 - beta^t because both start at 0
β1,β2\beta_1, \beta_2
usually 0.9 and 0.999
Worked by hand: two steps of each rule on one gradient stream (eta 0.1) Table
RuleStep 1 (g = 1)Step 2 (g = 3)State carried forward
SGD step0.1 x 1 = 0.1000.1 x 3 = 0.300none
Momentum stepv = 1, step 0.100v = 0.9 x 1 + 3 = 3.9, step 0.390velocity 3.9
Adam momentsm = 0.1, v = 0.001m = 0.39, v = 0.009999m = 0.39, v = 0.010
Adam bias-correctedm-hat = 1, v-hat = 1m-hat = 0.39 / 0.19 = 2.053; v-hat = 0.009999 / 0.001999 = 5.002-
Adam step0.1 x 1 / 1 = 0.1000.1 x 2.053 / sqrt(5.002) = 0.092-

Illustrative gradients for one weight of a retail demand-forecast network; Adam with beta1 0.9, beta2 0.999. When the gradient triples, SGD's step triples, momentum's grows further because the velocity carries the past, and Adam's barely moves: it divides by the gradient's running size. All values verified in numpy.

Adam's first step, exactly Sequence

  1. Moments

    m1 = 0.1 g and v1 = 0.001 g squared (both started at 0)

  2. Bias correction

    Divide by 1 - 0.9 = 0.1 and 1 - 0.999 = 0.001: m-hat = g, v-hat = g squared

  3. Step

    lr x g / |g| = lr x sign(g), whatever the size of g

  4. Here

    Gradients -1 and -0.75 both give a step of 0.1; the zero gradient gives 0

Later the denominator tracks each parameter's typical gradient size, so all parameters move at a similar relative pace.

Optimizer settings and their published defaults Table
SettingDefaultSourceWhat it trades
Adam learning rate0.001Kingma and Ba; scikit-learn learning_rate_initStep size; the first knob to sweep
Adam beta1, beta20.9, 0.999Kingma and Ba; scikit-learnMemory of the gradient and of its square
Adam epsilon1e-8Kingma and Ba; scikit-learnFloor for the denominator; matters when gradients are tiny
SGD momentum (scikit-learn solver sgd)0.9, Nesterov onscikit-learn 1.9.1Speed along consistent directions versus overshoot
PyTorch Adam weight_decay0 (an L2 term in the gradient)PyTorch Adam docsUse AdamW for decoupled decay (Day 34)

scikit-learn values read from the installed 1.9.1 MLPClassifier signature. With momentum beta, a steady gradient produces steps about 1 / (1 - beta) = 10 times the plain SGD step.

How the update rules move on a landscape3 minWhy SGD zig-zags, momentum speeds up and Adam rescales, read off the optimizer paths drawn in the concept block.
What each rule does to a narrow valley Overview

  • SGD: capped by the steepest side

    Stable only for a rate below 2 / largest curvature, so the flat direction gets tiny steps.

  • Momentum: velocity on the floor

    On a steady slope it moves about 1 / (1 - beta) = 10 times faster; too high a rate overshoots.

  • Adam: per-coordinate scale

    Divides by a running RMS of each coordinate's gradient, so steep and flat move at similar speeds.

Vanishing and exploding gradients3 minWhy the backward signal changes with depth and what keeps it in range.
Why the signal shrinks or grows with depth Overview

  • One factor per layer

    Each layer multiplies the backward signal by its weights and its local slopes.

  • Vanishing

    Sigmoid slopes are at most 0.25; ten of them can shrink the signal by orders of magnitude.

  • Exploding

    Large weights make the product grow; one step can send the loss to nan.

  • Remedies

    ReLU-family units, initialization matched to the activation (Day 34), clipping.

Gradient clipping by global norm Diagram
Compute allgradientsGlobal norm >threshold?Rescale to thresholddirection kept, lengthcappedOptimizer stepnoyesthen step
  1. Compute all gradients
  2. Global norm > threshold?
    • no: to Optimizer step
    yes
  3. Rescale to thresholddirection kept, length capped
    then step
  4. Optimizer step

Clipping treats exploding gradients (Pascanu et al., 2013). Check the learning rate and input scale first.

When training stalls, in this order3 minA decision flow from gradient check to optimizer comparison, and the four quantities worth logging.
Training stalls. Debug in this order Diagram
Gradient checkpasses?Fix the backwardstepbefore any tuningPer-layer normshealthy?Fix activation orinitor add clipping:architecture fixesSweep the learningratelog scale, per optimizerThen compareoptimizerseach at its best rate,several seedsnoyesnoyesthen
  1. Gradient check passes?
    • yes: to Per-layer norms healthy?
    no
  2. Fix the backward stepbefore any tuning
  3. Per-layer norms healthy?
    • yes: to Sweep the learning rate
    no
  4. Fix activation or initor add clipping: architecture fixes
    then
  5. Sweep the learning ratelog scale, per optimizer
  6. Then compare optimizerseach at its best rate, several seeds

Adam can train faster yet generalize worse than tuned SGD on some tasks (Wilson et al., 2017), so the last box is an experiment judged on validation data, not a rule.

What to log while training Overview

  • Loss

    Training and validation, per step or epoch; check the first value against ln K.

  • Gradient norm per layer

    A slope toward the input means vanishing; a spike or nan means exploding.

  • Update-to-weight ratio

    Step size relative to weight size per layer; very small means frozen, very large means unstable.

  • Non-finite values

    Stop on the first nan or inf and keep the batch that caused it.

The update-to-weight ratio is a practitioner diagnostic; thresholds quoted for it are rules of thumb, not results.

Memory anchor

Backprop finds the slope, the optimizer picks the step, and a gradient check says whether to trust the slope.

Useful for

  • Reading a training script - a churn model's, a demand forecaster's - and knowing which line computes gradients and which line changes the weights.

  • Debugging a custom layer or loss by comparing its gradient with a numerical one before any tuning.

  • Choosing what to log (per-layer gradient norms, update size relative to weight size) when training stalls.

Not suitable for

  • Deciding which optimizer will generalize best - that depends on data, architecture and tuning, and published comparisons disagree across tasks.

  • Verifying that the model or loss is the right one - a gradient check confirms the algebra of the code you wrote, not the design.

Traps

Trusting a falling loss as proof that gradients are right

A wrong backward step that is still roughly a descent direction lets the loss fall for a while, then it stalls well above what the model class can reach. The team tunes learning rates for days.

How to avoid it

Run a central-difference gradient check in float64 on a handful of examples whenever you write or change a backward pass.

Comparing optimizers at one learning rate

Adam and SGD with momentum have different good learning-rate ranges. At one shared value, one of them diverges or crawls, and the result is read as a verdict on the optimizer.

How to avoid it

Sweep the learning rate separately for each optimizer, with the same budget, and compare their best settings over several seeds.

Senior reasoning

A senior separates three questions that beginners blend: is the gradient correct (check it numerically), is the signal reaching every layer (look at per-layer norms), and is the update well chosen (learning rate, optimizer, schedule). They debug in that order, because tuning cannot rescue a wrong gradient and a better optimizer cannot rescue a saturated layer. They treat "Adam is better" or "SGD generalizes better" as hypotheses about a specific task, not rules.

Interview question

"Walk me through backpropagation for a two-layer network, and tell me what the optimizer does with the result." How would you answer?

Recall cards scheduled after completion

  • What does backpropagation compute, and what does the optimizer do with it?
  • Write the SGD, momentum and Adam updates in one line each, and say what Adam's first step looks like.

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

References

  • Learning representations by back-propagating errors — David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams, 1986core

    Presents back-propagation as repeatedly adjusting weights by gradients computed through hidden layers.

  • Dive into Deep Learning — Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola, 2023core

    Forward propagation, backward propagation and computational graphs, plus the optimization-algorithms chapter with momentum and Adam. (Chapter 5.3 (forward, backward and computational graphs) and Chapter 12 (optimization algorithms))

  • Adam: A Method for Stochastic Optimization — Diederik P. Kingma, Jimmy Ba, 2015optional

    Defines the Adam update with bias-corrected first and second moment estimates, the basis of the first-step behaviour shown today.

  • torch.autograd.gradcheck.gradcheck (PyTorch documentation) — PyTorchoptional

    Documents numerical-versus-analytic gradient checking and that its default tolerances assume double-precision inputs.

  • Learning long-term dependencies with gradient descent is difficult — Y. Bengio, P. Simard, P. Frasconi, 1994optional

    Formal account of why gradients vanish or explode when backpropagated through many steps.

  • Autograd mechanics — PyTorchoptional

    How autograd records the forward graph and saves tensors for the backward pass, the source of activation memory.

  • Adam (PyTorch documentation) — PyTorchoptional

    Documents that Adam's weight_decay is an L2 penalty with default 0, and the decoupled option equivalent to AdamW.

  • The Marginal Value of Adaptive Gradient Methods in Machine Learning — Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, Benjamin Recht, 2017optional

    Reports that adaptive methods can generalize worse than SGD on several deep-learning tasks, one reason optimizer choice is tested rather than assumed.

Optional sections

Deep dive · 10 min

Adam's first step, exactly. With m0=v0=0m_0 = v_0 = 0 and β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999: m1=0.1 gm_1 = 0.1\, g and v1=0.001 g2v_1 = 0.001\, g^2. The bias corrections divide by 1−β11=0.11 - \beta_1^1 = 0.1 and 1−β21=0.0011 - \beta_2^1 = 0.001, giving m^1=g\hat m_1 = g and v^1=g2\hat v_1 = g^2. The step is η g/(∣g∣+ϵ)≈η sign(g)\eta\, g / (|g| + \epsilon) \approx \eta\, \text{sign}(g), which is why the printed Adam update has equal entries (0.1, 0.1) for two parameters whose gradients are -1 and -0.75. Over many steps the denominator tracks the typical gradient magnitude, so every parameter moves at a similar relative pace.

Why central differences. (f(θ+ϵ)−f(θ−ϵ))/2ϵ(f(\theta + \epsilon) - f(\theta - \epsilon)) / 2\epsilon has error of order ϵ2\epsilon^2, while a one-sided difference has error of order ϵ\epsilon. Too small an ϵ\epsilon (below about 10−810^{-8}) hits floating-point cancellation, which is why the check is run in float64. Check a handful of parameters per layer on a few examples; as a practitioner rule of thumb, errors far above 10−510^{-5} in double precision deserve investigation.

Exploding gradients and clipping. When the backward signal grows, one step can throw the weights far away and the loss turns nan. Clipping rescales the gradient when its global norm exceeds a threshold (Pascanu et al., 2013), keeping the direction. It treats the symptom; check the learning rate and input scale first.

Cost of backpropagation. The backward pass costs a small multiple of a forward pass, and it needs the intermediates of the forward pass, which is why training memory scales with batch size and depth (Day 31's memory note, Day 40).

Hands-on lab: Gradient norms through depth, and one gradient under three update rules · 15 min

Run the block. The first part measures the gradient norm of each of 10 weight matrices after one forward and backward pass, for a sigmoid network and for a ReLU network with He initialization. The second part minimizes an ill-conditioned quadratic with SGD, momentum and Adam, each given its own best learning rate from the same small grid.

Predict before running: how many orders of magnitude separate layer 1 from layer 10 in the sigmoid network? Then answer: (1) Is the ReLU network's profile flat, and what part of that comes from the initialization? (2) The quadratic has the same exact gradient for every rule. Why do the three final values still differ? (3) This is one toy problem with one start. What would you need before saying which optimizer is better for a network?

import numpy as np

rng = np.random.default_rng(0)

# 1) Gradient norm per layer in a 10-layer, width-20 network: one forward and one backward pass
def layer_grad_norms(act, dact, scale, depth=10, width=20, batch=64):
    Ws = [rng.normal(0, scale, (width, width)) for _ in range(depth)]
    a, zs, As = rng.normal(size=(batch, width)), [], []
    for W in Ws:
        As.append(a); z = a @ W; zs.append(z); a = act(z)
    delta, norms = (a - 1.0) / batch, []
    for W, z, A in zip(Ws[::-1], zs[::-1], As[::-1]):
        delta = delta * dact(z)
        norms.append(np.linalg.norm(A.T @ delta))   # gradient of this layer's weight matrix
        delta = delta @ W.T
    return norms[::-1]                              # layer 1 first

sig = lambda z: 1 / (1 + np.exp(-z))
for label, act, dact, scale in [
    ("sigmoid, N(0, 0.5)   ", sig, lambda z: sig(z) * (1 - sig(z)), 0.5),
    ("ReLU, He init        ", lambda z: np.maximum(0, z), lambda z: (z > 0) * 1.0, np.sqrt(2 / 20)),
]:
    n = layer_grad_norms(act, dact, scale)
    print(f"{label} layer 1: {n[0]:.2e}  layer 5: {n[4]:.2e}  layer 10: {n[9]:.2e}  ratio 10/1: {n[9] / n[0]:.1e}")

# 2) Same gradient, different update rules on an ill-conditioned quadratic f(w) = 0.5 * (w1^2 + 25 * w2^2)
curv = np.array([1.0, 25.0])
f = lambda w: 0.5 * np.sum(curv * w ** 2)
def run(rule, lr, steps=40):
    w, v, m, s = np.array([3.0, 3.0]), np.zeros(2), np.zeros(2), np.zeros(2)
    for t in range(1, steps + 1):
        g = curv * w                                # exact gradient: identical for every rule
        if rule == "sgd":
            w = w - lr * g
        elif rule == "momentum":
            v = 0.9 * v + g; w = w - lr * v
        else:                                       # Adam: bias-corrected moment estimates
            m = 0.9 * m + 0.1 * g; s = 0.999 * s + 0.001 * g ** 2
            w = w - lr * (m / (1 - 0.9 ** t)) / (np.sqrt(s / (1 - 0.999 ** t)) + 1e-8)
    return f(w)

grid = [0.5, 0.2, 0.1, 0.05, 0.02, 0.01, 0.005]
for rule in ("sgd", "momentum", "adam"):
    best = min((run(rule, lr), lr) for lr in grid)
    print(f"{rule:>8}: best lr on the grid {best[1]}, f after 40 steps = {best[0]:.2e}")
Interview practice · 10 min

A colleague on a media recommender team says "Adam fixes vanishing gradients because it adapts the learning rate per parameter." How do you respond?

Content version 6 · last verified 2026-10-18 · depends on numpy