Core day 34 of 60 · Week 7 · Part II · application depth

Training Deep Networks

A digit classifier's training loss is 0.003, but its validation loss bottomed out at epoch 120 and has crept up since. Another run's loss jumps to 24 on the first update. For each curve, which single change do you make first, and what do you check afterwards?

By the end: From per-epoch training and validation losses you label a run diverging, underfitting, overfitting or healthy, cite the epochs that support the label, and choose one intervention whose effect you can verify on the next run.

At a glance

Read training and validation loss together, name the symptom, change one knob that targets it, and rerun with the same seed and split.

Diverging
Loss jumps or turns nan: lower the learning rate, check input scale, warm up.
Underfitting
Both curves high and slow: raise the rate, train longer, add capacity.
Overfitting
Training falls, validation turns up: stop at the minimum, then regularize.
Scale at step 0
Initialization matched to the activation keeps signals near size 1 through depth.
One change per run
Same seed and split; a bundle of changes cannot be attributed.

Recall · 5 min

Reviews due from Days 15, 32 and 33 appear first. If none are due: write one sentence separating backpropagation from the optimizer; say what the loss of a 10-class softmax network should be at initialization and why; and describe in one line how early stopping worked for gradient boosting on Day 15.

Open due reviews

Concept · 10 min

Deep networks fail in a few recognizable ways, and the two loss curves together say which: diverging, underfitting, overfitting or healthy. Each symptom has one first intervention. Change one thing per run, or you cannot tell what helped; practitioner checklists such as Karpathy's "recipe" say the same.

Each symptom has its own curve shape, and the shape tells you which single knob to turn

Predict first

Training loss is 0.01 and falling; validation loss fell until epoch 20 and has risen steadily since. What is the lowest-risk first change?

Text description of the figure

Training and validation loss curves for a 2-16-16-1 tanh network trained live in the browser on 40 noisy two-moons points (not the digits of the code block), redrawn as you change learning rate, weight decay, dropout and epochs. A high learning rate gives a jagged or exploding curve, a tiny one a slow flat curve, and a long run on little data shows the training loss falling while the validation loss turns upward.

Read the two curves together Map

Overfitting

Memorizing. Stop at the validation minimum, then add decay or dropout, or more data.

Suspicious

Both high and far apart: check the split, preprocessing parity and label quality.

Healthy

Both low and close. Confirm over seeds; consider more capacity only if it pays.

Underfitting

Both high and close: raise the rate, train longer, or add capacity.

Divergence is the fifth case: a training curve that jumps up or turns nan. Lower the learning rate first.

Four runs of the same small network, one setting changed each time Illustration

a tiny step: training 0.50, validation 0.53 at the end; validation best 0.53 at epoch 120.

a huge step: training 1.13, validation 3.16 at the end; validation best 0.57 at epoch 0.

a sensible step, long run: training 0.10, validation 0.61 at the end; validation best 0.41 at epoch 9.

the same run, regularized: training 0.35, validation 0.45 at the end; validation best 0.40 at epoch 69.

training loss
validation loss (circle: its minimum)
dashed ln 2 = 0.693: a coin-flip guess

Computed: the training-curves widget's network and data (40 training, 60 validation points, batches of 8, seed 1), curves clipped at 1.4. The label is the widget's rule-based diagnosis, not a hand annotation.

The label on each panel is the training-curves widget's own rule, applied to the computed curves. Read the pair of curves, not either one alone.

Measured curves from today's run: learning rate 0.5 versus 0.01 Chart

lr 0.5, training
lr 0.5, validation
lr 0.01, training
lr 0.01, validation

Values recorded from the application code. lr 0.5: validation bottoms at 0.475 and creeps to 0.520 (overfitting). lr 0.01: both still falling at epoch 600 (underfitting).

One symptom, one intervention, one check Table

Symptom	Likely cause	First intervention	Check on the next run
Loss explodes or nan	Rate too high, unscaled inputs	Lower the rate 3-10x; warmup or clipping	No spikes in training loss
Both curves flat and high	Rate too low, too few epochs	Raise the rate or train longer	A tiny batch can be fit exactly
Train low, validation rising	Memorization	Early stopping at the validation minimum	Final validation equals the minimum
Deep network does not start	Activations vanish or explode	Initialization matched to the activation	Per-layer activation scale near 1

The debugging loop: one change per lap Loop

Plot both curvestraining and validation, same axesread the pair ↓
Name the symptomdiverging, under, over or healthypick its knob ↓
Change one knobthe one the symptom points torerun ↓
Rerun and compareagainst the seed spreadnext lap ↓
↺ back to 1 · Plot both curves

Keep and log a change only if it moves the curve beyond seed-to-seed variation; otherwise revert it. Reverting means the symptom was probably misread: go back to the pair of curves, not to a new knob.

The knobs and what each one fixes Overview

Initialization
Sets activation and gradient scale at step 0; match it to the activation.
Normalization
BatchNorm or LayerNorm keeps each layer's inputs near zero mean, unit scale.
Dropout
Zeroes random units in training only; off in evaluation.
Weight decay
Shrinks weights each step; with Adam use the decoupled AdamW form.
Early stopping
Keeps the weights from the validation minimum.
Schedule
Warmup starts gently, decay ends with small steps.

Scale at step 0: Weight initialization matched to the activation (Glorot and Bengio, 2010; He et al., 2015 for ReLU).
Normalization: Batch normalization (per feature, over the batch) and Layer normalization (per example, over its features).
Regularization: Dropout (training only), Weight decay (with Adam use AdamW; Loshchilov and Hutter, 2019) and Early stopping.
Step size over time: a Learning-rate schedule, warmup then decay.
The pictures map each symptom to its knob and the check that confirms it.

Assumptions

Training and validation rows come from the same process, and validation only monitors and picks the stopping point; once it guides many decisions its loss is optimistic (Days 9, 15).
The curves come from one seed; seed-to-seed variation can match a small intervention's effect.
Loss is the monitored quantity; accuracy can stay flat while validation loss rises as the model grows overconfident (Day 32).
Regularizers trade variance for bias; too much decay or dropout underfits.

Application and workplace decision · 10 min

Same 64-64-10 ReLU network, 60 training digits, 400 validation digits (scikit-learn's bundled digits, rescaled by a fixed 1/16), full-batch gradient descent for 600 epochs, four settings:

learning rate 0.5;
learning rate 0.5 with weight decay 0.001;
learning rate 0.01;
learning rate 40.

Write diverging, underfitting, overfitting or healthy for each before running. The block prints the training loss after the first update and at the end, the best validation loss with its epoch, and the last validation loss.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
from sklearn.datasets import load_digits

X, y = load_digits(return_X_y=True)
X, Y = X / 16.0, np.eye(10)[y]                    # pixels are 0-16: a fixed rescale, nothing fitted
perm = np.random.default_rng(0).permutation(len(X))
tr, va = perm[:60], perm[1000:1400]               # tiny training set on purpose; other rows stay unused

def train(lr, wd=0.0, epochs=600, h=64):
    r = np.random.default_rng(0)
    W1, b1 = r.normal(0, np.sqrt(2 / 64), (64, h)), np.zeros(h)
    W2, b2, hist = r.normal(0, np.sqrt(1 / h), (h, 10)), np.zeros(10), []
    def fwd(A):
        Z = np.maximum(0, A @ W1 + b1) @ W2 + b2
        E = np.exp(Z - Z.max(1, keepdims=True))
        return E / E.sum(1, keepdims=True)
    ce = lambda rows: -np.mean(np.sum(Y[rows] * np.log(fwd(X[rows]) + 1e-12), axis=1))
    for _ in range(epochs):
        A = X[tr]
        H = np.maximum(0, A @ W1 + b1)
        dZ = (fwd(A) - Y[tr]) / len(A)                         # softmax + cross-entropy gradient
        dH = (dZ @ W2.T) * (H > 0)
        W2, b2 = W2 - lr * (H.T @ dZ + wd * W2), b2 - lr * dZ.sum(0)
        W1, b1 = W1 - lr * (A.T @ dH + wd * W1), b1 - lr * dH.sum(0)
        hist.append((ce(tr), ce(va)))
    return np.array(hist)

for name, kw in [("lr 0.5", dict(lr=0.5)), ("lr 0.5 + weight decay", dict(lr=0.5, wd=0.001)),
                 ("lr 0.01", dict(lr=0.01)), ("lr 40", dict(lr=40.0))]:
    h = train(**kw)
    b = h[:, 1].argmin()
    print(f"{name:<22} train {h[0,0]:.2f} -> {h[-1,0]:.3f} | val best {h[b,1]:.3f} at epoch {b+1}, last {h[-1,1]:.3f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 26 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

Why there is a cliff: gradient descent on a quadratic4 minThe exact update factor, the 2/lambda threshold derived, four rates worked by hand, and the same cliff on the real network.

Gradient descent on a quadratic: the whole dynamics in one factor Formula

w_{t+1} = w_t - \eta\, \lambda w_t = (1 - \eta\lambda)\, w_t, \qquad w_t = (1 - \eta\lambda)^t\, w_0

$f(w) = \tfrac{1}{2}\lambda w^2$: the model problem; lambda is the curvature and the gradient is lambda w
$1 - \eta\lambda$: each step multiplies w by this fixed factor, so the iterates are a geometric sequence
$|1 - \eta\lambda| < 1$: the convergence condition; solving it gives 0 < eta < 2 / lambda
$2 / \lambda_{max}$: with several directions, the steepest curvature sets the ceiling for the shared rate

Between 1/lambda and 2/lambda the factor is negative: the step overshoots the minimum yet still converges. Past 2/lambda every overshoot grows. Near a minimum a real loss is approximately quadratic, with lambda its largest curvature: that is why every learning-rate sweep shows a gentle slope, then a cliff.

Worked by hand: lambda = 25, w0 = 1, so the threshold is 2/25 = 0.08 Table

Rate	Factor 1 - 25 eta	w over steps 0-4	Behaviour
0.01	+0.75	1, 0.75, 0.563, 0.422, 0.316	Smooth decay
0.07	-0.75	1, -0.75, 0.563, -0.422, 0.316	Overshoots, still converges
0.08	-1.00	1, -1, 1, -1, 1	The edge: oscillates forever
0.09	-1.25	1, -1.25, 1.563, -1.953, 2.441	Diverges

Each row is the geometric sequence (1 - eta lambda)^t, verified in numpy. The loss at rate 0.09 grows by 1.25 squared = 1.5625 per step.

The derivation, drawn: three rates around the threshold Illustration

Well below the threshold 2/λ = 0.08: each step keeps 75% of the distance, so w decays smoothly (1, 0.75, 0.5625, …) and the loss falls every step.

Past η = 1/λ the factor turns negative: the step overshoots the minimum each time (1, −0.75, 0.5625, …), but |factor| < 1, so the oscillation still dies out.

Above 2/λ = 0.08 the factor's size exceeds 1: every overshoot lands farther out (1, −1.25, 1.5625, −1.9531, …) and the loss grows without bound.

Computed exactly: gradient descent w ← w − η·λ·w on f(w) = ½·λ·w² with λ = 25 and w₀ = 1, for the three printed rates. The iterates are the geometric sequence (1 − ηλ)ᵗ; no simulation noise.

Below the threshold the iterates decay; past 1/lambda they overshoot but still shrink; above 2/lambda every overshoot lands farther out. The lr 40 run on the digits network is the third panel's behaviour in the wild.

Initialization, derived and measured4 minThe variance argument behind Glorot and He, why ReLU needs twice the variance, and what a badly scaled start does to 20 layers.

The He and Glorot variances, derived Sequence

One pre-activation
z = sum of w_i x_i over n_in independent zero-mean terms, so Var(z) = n_in x Var(w) x Var(x)
Keep the scale
Asking Var(z) = Var(x) forces Var(w) = 1 / n_in: the Glorot-style forward condition for tanh-like units
ReLU halves it
ReLU zeroes the negative half of a symmetric z, so the second moment of its output is Var(z) / 2; compensating needs Var(w) = 2 / n_in (He)
Backward pass
The same argument on the gradients gives Var(w) = 1 / n_out; Glorot's published compromise is 2 / (n_in + n_out)
Measured
The table below shows what 20 ReLU layers actually do to the activation scale under each rule

The derivation assumes independent zero-mean weights and inputs and holds at initialization only; training then moves the weights.

Initialization variance by activation Formula

\operatorname{Var}(W) = \frac{1}{n_{in}} \;\text{(tanh-like, Glorot-style)}, \qquad \operatorname{Var}(W) = \frac{2}{n_{in}} \;\text{(ReLU, He)}

$n_{in}$: number of inputs to the layer (fan-in)
$2$: ReLU zeroes about half its inputs, so it needs twice the variance to keep the scale

Glorot and Bengio also give 2 / (n_in + n_out), which balances the forward and backward passes.

The initial weight scale, before any training Illustration

Size after 20 ReLU layers: 8e-29 (sd 0.01), 7e-4 (Var 1/n), 0.76 (He), 8e5 (sd 0.5). Dashed: the prediction (n·Var(w)/2)^(l/2), which the measured lines follow.

Tanh, product of the mean slope over the layers so far (what a backward signal keeps): 1e-1 with Var 1/n, 3e-4 with Var 2/n, 2e-10 with sd 0.5. Too large a scale saturates tanh units.

N(0, 0.01) (ReLU panel only)
Var 1/n (Glorot-style)
Var 2/n (He)
N(0, 0.5)

Computed: one seeded batch of 16 standard-normal rows through 20 freshly drawn 32 x 32 weight matrices, no biases, no training. n = 32 inputs per layer here, so He means sd √(2/32) = 0.25.

In a ReLU stack each layer multiplies the activation size by about the square root of n Var(w) / 2, so only Var = 2/n stays level. In a tanh stack a large scale saturates the units and the backward signal collapses.

Std of the activations after 20 ReLU layers (width 100), by initialization Table

Initialization	Weight std	Std at layer 20	Verdict
N(0, 0.01)	0.01	4.6e-24	Signal vanished
Glorot-style, Var 1/n	0.1	4.4e-04	Shrinking: too small for ReLU
He, Var 2/n	0.141	2.0	Stable
N(0, 0.3)	0.3	2.8e+06	Exploded

Values printed by this lesson's hands-on lab (forward pass only, one draw of weights). Library defaults differ: scikit-learn's MLP uses a Glorot-style uniform range even for ReLU units (Day 31), and normalization layers soften a poor start.

The learning rate, the batch and the schedule4 minA log-scale sweep, how batch size moves the best rate, and warmup then cosine decay.

Two ways to find the cliff before you tune anything else Comparison

Criterion	Sweep of separate runs	Range test (Smith, 2017)
How it runs	One short run per rate, rates spaced on a log scale	One short run, the rate raised a little every step
Cost	k runs for k rates	1 run
What you read off	Best final loss per rate; the edge where runs start to fail	The rate where the loss stops falling and turns up

Both answer the same question: where the gentle slope ends. The range test is cheaper; separate runs also show how each rate ends, which the sweep below draws.

Too small, workable, too large Illustration

rate 0.001: 0.49, rate 0.3: 0.34, rate 3: 1.43 training loss after 80 epochs.

Lowest training loss at rate 1, lowest validation loss at rate 0.3. The loss falls slowly as the rate grows, then jumps off a cliff: rate 10 diverges (×). Sweep rates on a log scale (or run a range test that raises the rate within one run) before tuning anything else.

rate 0.001: too small
rate 0.3: workable
rate 3: too large
right: final training loss
right: best validation loss (dashed)

Computed: the training-curves widget's data and 2-16-16-1 tanh network, minibatch SGD with batches of 8, seed 1, 80 epochs per rate; loss clipped at 1.6. One seed: the exact best rate moves between seeds, the U shape does not.

The loss improves slowly as the rate rises, then falls off a cliff. Sweep on a log scale; the best rate sits just below the edge. Momentum and Adam move the stable range and the speed inside it; they do not remove the cliff (Day 33's lab measures all three on the quadratic).

Batch size and learning rate move together Overview

Linear scaling rule
Multiply the batch by k, multiply the rate by k, with a warmup (Goyal et al., 2017, for large-batch SGD).
Gradient noise
Small batches give noisier gradients: more updates per epoch, each less precise.
BatchNorm statistics
Batch size also changes the statistics BatchNorm normalizes with; very small batches make them noisy.
Re-tune after a change
A new batch size invalidates the old best rate; sweep again before comparing.

Warmup for 5 steps, then cosine decay to 0 over 100 steps (peak 0.1) Chart

Exact values of the lab's schedule function. Warmup avoids large early steps while Adam's moments and normalization statistics settle. The plain cosine is the single-cycle case of cosine decay with restarts (SGDR, Loshchilov and Hutter, 2017).

BatchNorm and LayerNorm4 minOne formula on two axes, both computed by hand on three rows, and why the axis matters at batch size 1 and in evaluation.

BatchNorm and LayerNorm are the same formula on different axes Formula

\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}}, \qquad y = \gamma\, \hat{x} + \beta

$\mu, \sigma^2$: BatchNorm: per feature, across the batch rows. LayerNorm: per example, across its own features
$\epsilon$: a small floor (about 1e-5) so the division never blows up on a constant feature
$\gamma, \beta$: learned scale and shift per feature, so the layer can undo the normalization where that helps

Only the axis of the statistics differs, and that one choice drives everything in the comparison below, from batch-size sensitivity to the evaluation-mode switch.

Worked by hand: BatchNorm and LayerNorm on three sensor rows (manufacturing) Table

Row	Input (temp, vibration, pressure)	BatchNorm: per column	LayerNorm: per row
1	(1, 2, 3)	(-1.225, -1.225, -1.389)	(-1.225, 0, 1.225)
2	(3, 4, 8)	(0, 0, 0.926)	(-0.926, -0.463, 1.389)
3	(5, 6, 7)	(1.225, 1.225, 0.463)	(-1.225, 0, 1.225)
Statistics	-	column means (3, 4, 6), variances (2.667, 2.667, 4.667)	row means (2, 5, 6), variances (0.667, 4.667, 0.667)
Row 1 alone (batch of 1)	(1, 2, 3)	(0, 0, 0): each value minus itself	(-1.225, 0, 1.225): unchanged

Illustrative production-line readings. Variances divide by n, the biased form both layers normalize with; epsilon = 1e-5 is negligible here, gamma = 1 and beta = 0. Rows 1 and 3 differ only by a shift of 4, so LayerNorm gives them identical outputs: it discards each row's level. Verified in numpy.

BatchNorm versus LayerNorm Comparison

Criterion	BatchNorm	LayerNorm
Normalizes over	Each feature, across the rows of the mini-batch	Each example, across its own features
Depends on batch size	Yes; small batches give noisy statistics	No
Training versus evaluation	Batch statistics in training, running averages in evaluation	Same computation in both modes
Batch of 1	No statistics: zeros in training mode; PyTorch BatchNorm1d raises an error on (1, C)	Works normally
Typical home	Convolutional networks (Day 36)	Transformers and recurrent networks (Days 37, 39)

BatchNorm keeps two sets of statistics, and the mode picks one Diagram

Training batchmodel.train()
- to Batch mean and var
Evaluation inputmodel.eval(), any batch size
- normalize: to Normalize, fixed stats
Weights averagedSWA or EMA: stats now stale
- to update_bn pass
Batch mean and varnormalize this batch with them
- momentum 0.1: to Running estimates
update_bn passone pass over training data
recompute
Running estimatesone mean and var per feature
read
Normalize, fixed statsindependent of the other rows

PyTorch's BatchNorm updates its running estimates with momentum 0.1 during training and reads them in evaluation (BatchNorm2d docs). Averaged weights (torch.optim.swa_utils) never produced those statistics, so recompute them before trusting evaluation.

Dropout and weight decay, done right3 minInverted dropout's rescaling, and why Adam needs the decoupled decay of AdamW.

Inverted dropout, in training and evaluation Sequence

Training
Zero each unit with probability p, for example 0.5
Rescale
Divide the survivors by 1 - p so the expected activation is unchanged
Evaluation
Turn dropout off; no rescaling needed (call model.eval() in PyTorch)
Check
Lab: training mean 0.999 versus evaluation mean 1.000

L2 penalty inside Adam versus decoupled weight decay (AdamW) Comparison

Criterion	L2 inside the gradient	AdamW
Where the decay enters	lambda times theta is added to g before Adam's moments	Subtracted from the weights directly, outside the moments
Effect of the adaptive step	Divided by sqrt(v-hat): weights with large gradients are decayed less	Every weight shrinks by the same fraction lr times lambda
Result	Uneven, hard-to-tune regularization	Decay behaves like plain weight decay in SGD

Loshchilov and Hutter (2019). For plain SGD the two forms coincide up to a rescaling of lambda.

Early stopping as a regularizer2 minThe knobs, their scikit-learn defaults, and what to write yourself in PyTorch.

Early-stopping knobs, and scikit-learn's defaults (1.9.1, checked) Table

Knob	MLPClassifier default	What it controls
early_stopping	False	Off unless switched on: training runs to max_iter or the tol rule
validation_fraction	0.1	Share of training rows held out to monitor
n_iter_no_change (patience)	10	Epochs without improvement before stopping
tol (minimum improvement)	0.0001	Smaller changes count as no improvement
Best weights	restored when early_stopping is on	The coefficients with the best validation score are kept

In plain PyTorch you write this loop yourself: track the best validation loss, checkpoint it, stop when patience runs out, reload the checkpoint. The rows that pick the stopping epoch are now tuning rows, so the final estimate needs a separate test part (Days 9 and 15).

A checklist before tuning2 minFive checks that find bugs before you spend runs on knobs.

Before tuning anything Sequence

Check the loss at step 0
About ln K for K balanced classes: ln 10 = 2.30 for digits
Overfit one small batch
If it cannot reach near-zero loss, look for a bug, not a knob
Simplest model first
No dropout, no decay, a known-good learning rate
Add one thing at a time
Each with its own run, seed and logged curves
Then regularize
Only after the training curve shows enough capacity

A practitioner checklist in the spirit of Karpathy's recipe (2019).

Run hygiene and reproducibilityadvanced3 minWhat a seed does and does not guarantee, what a checkpoint must hold, and two ordering bugs that bend curves.

Run hygiene that makes curves comparable Overview

Seeds are not enough
PyTorch does not promise identical results across releases, platforms or CPU versus GPU.
Checkpoint everything
Model and optimizer state (Adam's moments) plus epoch, to resume a run honestly.
Scheduler order
Call scheduler.step() after optimizer.step(); the reverse skips the first rate.
Mode test
Same input twice in eval mode must give the same output; catches dropout or BatchNorm left in training mode.
Seed spread
Report the spread over three or more seeds next to any claimed improvement; one seed is one sample (Day 35).

From the PyTorch reproducibility, saving-and-loading and optim documentation (2.14 pages).

Memory anchor

Read both curves, name the symptom, change one thing, and rerun with the same seed and split.

Useful for

Triage of a failed or unstable training run - a demand forecaster's, a defect classifier's, a churn model's - from the logged loss curves before touching hyperparameters.
Writing a run log that states the symptom, the single change made, and the result against a stated expectation.
Explaining to a stakeholder why a model was stopped early or why a larger network was not the answer.

Not suitable for

Deciding whether the model is good enough to ship - that needs the sealed test evaluation and the cost of errors (Days 16 and 35).
Fixing data problems such as label noise, leakage or a shifted validation set; regularization can only hide them.

Traps

Reading overfitting as divergence, or the reverse

A rising validation loss is called "divergence" and the learning rate is lowered, although the training loss is still falling. Or an exploding run is called overfitting and regularizers are added.

How to avoid it

Always read training and validation loss together; divergence shows in the training curve, overfitting in the gap.

Fixing everything at once

Learning rate, weight decay, dropout, schedule and batch size change in one run. The curve improves, nobody knows why, and the next model starts from scratch.

How to avoid it

Change one thing per run, pre-register what you expect, and use several seeds when the difference is small.

Evaluating in training mode

Dropout stays active and BatchNorm uses batch statistics, so validation loss fluctuates between runs and depends on batch composition.

How to avoid it

Switch to evaluation mode before validation and serving, and add a test that the same input gives the same output twice.

Senior reasoning

A senior treats the training log as evidence and the intervention as a hypothesis. They first confirm the basics (loss at initialization near $\ln K$ , a tiny batch can be overfit, input scale is sane), then name the symptom, change one thing, and state beforehand what the curves should do. They know a single seed is a sample, so small differences are repeated before they are believed, and they remember that regularization trades variance for bias: if the validation minimum itself is poor, the next step is data or capacity, not more penalty.

Interview question

"Training loss keeps falling but validation loss has started rising. What do you do?" How would you answer?

Outline your answer first.

Recall cards scheduled after completion

Name the four curve patterns and the single first intervention for each.
How do dropout and BatchNorm behave differently in training and evaluation mode?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Understanding the difficulty of training deep feedforward neural networks — Xavier Glorot, Yoshua Bengio, 2010core
Analyses how activations and gradients behave across layers at initialization and proposes a variance-balancing scheme for tanh-like activations.
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification — Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, 2015optional
Derives the initialization variance for ReLU networks (twice the Glorot-style value for the same fan-in).
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — Sergey Ioffe, Christian Szegedy, 2015optional
Defines batch normalization with mini-batch statistics in training and population estimates at inference; the paper's covariate-shift explanation is the authors' motivation and not settled fact.
Layer Normalization — Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton, 2016optional
Defines layer normalization over the features of one example, independent of batch size.
Dropout: A Simple Way to Prevent Neural Networks from Overfitting — Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov, 2014optional
Presents dropout as random unit removal in training with a matched approximation at test time.
Decoupled Weight Decay Regularization — Ilya Loshchilov, Frank Hutter, 2019core
Shows that L2 regularization and weight decay differ for adaptive optimizers and proposes decoupled weight decay (AdamW).
Dropout (PyTorch documentation) — PyTorchoptional
States that dropout rescales by 1/(1-p) in training and computes the identity in evaluation.
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour — Priya Goyal, Piotr Dollar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, Kaiming He, 2017optional
The linear scaling rule for learning rate versus minibatch size, with a gradual warmup.
Cyclical Learning Rates for Training Neural Networks — Leslie N. Smith, 2017optional
The learning-rate range test for finding reasonable learning-rate bounds in a short run.
SGDR: Stochastic Gradient Descent with Warm Restarts — Ilya Loshchilov, Frank Hutter, 2017optional
Cosine annealing with warm restarts, the origin of the cosine schedule used in practice.
Reproducibility (PyTorch notes) — PyTorch contributorsoptional
States that identical seeds do not guarantee identical results across releases, platforms or CPU and GPU.
Saving and loading models, including general checkpoints (PyTorch tutorials) — PyTorchoptional
A resumable checkpoint stores the model and the optimizer state dicts plus the epoch.
torch.optim (PyTorch documentation) — PyTorchoptional
Scheduler.step() is called after optimizer.step(); the reverse order skips the first value of the schedule.
BatchNorm2d (PyTorch documentation) — PyTorchoptional
BatchNorm keeps running mean and variance estimates (momentum 0.1) in training and uses them in evaluation.
A Recipe for Training Neural Networks — Andrej Karpathy, 2019optional
Practitioner checklist (loss at initialization, overfit one batch, add complexity one step at a time); cited as experience, not as an empirical study.

Optional sections

Deep dive · 10 min

Why initialization needs a scale. For $y = \sum_{i=1}^{n} w_i x_i$ with independent zero-mean terms, $\text{Var}(y) = n\, \text{Var}(w)\, \text{Var}(x)$ . Choosing $\text{Var}(w) = 1/n$ keeps the variance constant from layer to layer for linear or tanh-like units. ReLU sets about half of its inputs to zero, halving the second moment, so $\text{Var}(w) = 2/n$ compensates (He et al.). The lab prints the result: after 20 ReLU layers, $2/n$ keeps the activation scale near 2, while $1/n$ shrinks it to about $4 \times 10^{-4}$ and a standard deviation of 0.3 inflates it to millions.

BatchNorm versus LayerNorm. BatchNorm computes mean and variance per feature across the batch, so a batch of one has no information and small batches give noisy statistics; at inference it substitutes running averages. LayerNorm computes them per example across features, which is why Transformer blocks use it (Day 39): same computation in training and evaluation, any batch size.

Weight decay versus L2. For plain SGD, adding $\lambda \|w\|^2 / 2$ to the loss is the same as shrinking each weight by a factor $(1 - \eta \lambda)$ per step. For Adam, an L2 term enters the gradient and is then divided by the adaptive denominator, so weights with large gradient history are penalized less. AdamW applies the shrinkage directly, outside the adaptive step. Exclude biases and normalization scales from decay as a common practice.

Schedules. Warmup ramps the rate up over the first steps (a common explanation is that Adam's second-moment estimates are unreliable at the start); cosine decay lowers it smoothly to near zero. A learning-rate range test (raise the rate exponentially for a few hundred steps and watch where the loss turns up) gives a starting value for the peak.

Hands-on lab: Initialization, normalization, dropout and a schedule, one at a time · 15 min

Run the block. Part 1 pushes a batch through 20 ReLU layers with four weight scales and prints the activation scale at the end. Part 2 contrasts batch and layer normalization on a batch and on a single row. Part 3 checks inverted dropout. Part 4 prints a warmup-plus-cosine schedule.

Predict first: which of the four initializations keeps the final activation scale within a factor of ten of the input's? Then answer: (1) Why does the Glorot-style variance shrink the signal in a ReLU network, and what would you use for tanh? (2) What does BatchNorm output for a batch of one, and why does that matter at serving time? (3) Why is no correction needed at evaluation time with inverted dropout? (4) Where in a training loop would you call the schedule?

import numpy as np

rng = np.random.default_rng(0)
width, depth = 100, 20
X0 = rng.normal(size=(256, width))

# 1) Activation scale after 20 ReLU layers under different initializations (forward pass only)
def last_layer_std(scale):
    a = X0
    for _ in range(depth):
        a = np.maximum(0, a @ rng.normal(0, scale, (width, width)))
    return a.std()
for name, s in [("N(0, 0.01)", 0.01), ("Glorot-style, Var 1/n", np.sqrt(1 / width)),
                ("He, Var 2/n", np.sqrt(2 / width)), ("N(0, 0.3)", 0.3)]:
    print(f"init {name:<24} std of layer-{depth} activations: {last_layer_std(s):.3e}")

# 2) BatchNorm normalizes each feature across the batch; LayerNorm normalizes each example across its features
def batchnorm(a, eps=1e-5):
    return (a - a.mean(0)) / np.sqrt(a.var(0) + eps)
def layernorm(a, eps=1e-5):
    return (a - a.mean(1, keepdims=True)) / np.sqrt(a.var(1, keepdims=True) + eps)
A = rng.normal(5.0, 3.0, size=(8, 4))
print("BN column means", batchnorm(A).mean(0).round(6), "| LN row means", layernorm(A).mean(1).round(6))
one = A[:1]
print("batch of 1: BN output", batchnorm(one).round(3), "(all zeros, no information) | LN output", layernorm(one).round(2))

# 3) Inverted dropout: scale by 1/(1-p) in training so evaluation needs no change
p = 0.5
a = np.ones((10000, 10))
mask = rng.random(a.shape) >= p
print(f"train mean {(a * mask / (1 - p)).mean():.3f} (expected 1.000) | eval mean {a.mean():.3f} (identity)")

# 4) Learning-rate schedule: 5 warmup steps, then cosine decay to zero over 100 steps
def lr_at(t, peak=0.1, warm=5, total=100):
    if t < warm:
        return peak * (t + 1) / warm
    return 0.5 * peak * (1 + np.cos(np.pi * (t - warm) / (total - warm)))
print("lr at steps 0, 4, 5, 50, 99:", [round(float(lr_at(t)), 4) for t in (0, 4, 5, 50, 99)])

Interview practice · 10 min

On an insurance claims-fraud classifier, your validation loss is flat and high from the first epoch, while the training loss is also high and barely moving. A teammate wants to add dropout. What do you ask and say?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-18 · depends on numpy, scikit-learn

1 Recall · 5 min

2 Concept · 10 min

Assumptions

3 Application and workplace decision · 10 min

4 Mini-assessment · 8 min

Technical depth · optional · 26 min in total

Useful for

Not suitable for

Traps

Senior reasoning

Interview question

Recall cards scheduled after completion

Prerequisites and related concepts

References

Optional sections

Recall · 5 min

Concept · 10 min

Application and workplace decision · 10 min

Mini-assessment · 8 min