Core day 32 of 60 · Week 7 · Part II · application depth

Activations and Losses

A card-fraud network ends with a raw score of -2.0 for a transaction that did turn out to be fraud (label 1). Is that score a probability, which loss should train it, and why does a squared-error loss barely move the weights when the network is confidently wrong?

By the end: Given a prediction task you choose the output layer and the loss, explain why the loss should receive logits, and compute the gradient with respect to the logit for cross-entropy and for squared error on a small example.

At a glance

The activation's slope decides how much signal flows back, the output layer follows the label, and the loss should read logits, not probabilities.

Activations bend
Sigmoid and tanh saturate in the tails; ReLU's slope is 0 or 1; GELU is a smooth ReLU.
Logits first
The network outputs logits; sigmoid or softmax makes probabilities only when needed.
Label picks the output
One of K: softmax. Several of K: K sigmoids. A number: one linear unit.
Cross-entropy
Gradient p - y stays large when the model is confidently wrong.
Stable softmax
Subtract the max logit; losses that take logits do this inside.

Recall · 5 min

Reviews due from Days 11, 13 and 31 appear first. If none are due: write the shape of every array in the 2-3-1 network for a batch of 4, then say in one sentence why removing the ReLU turns it into a linear model. From Day 11: what does logistic regression minimize, and what is its output called before the sigmoid?

Open due reviews

Concept · 10 min

A hidden unit sums, then bends. The bend is the Activation function, and its slope matters as much as its shape, because Day 33 multiplies slopes along the network. The network's raw output is a Logit, any real number; a sigmoid or Softmax turns logits into probabilities, and the loss should read the logits.

Squared error on a sigmoid goes quiet exactly where the prediction is most wrong; cross-entropy does not

Predict first

A sigmoid output is at p = 0.01 for a true label y = 1. Which loss gives the larger gradient magnitude with respect to the logit?

Text description of the figure

A panel with an activation selector, a loss selector and a logit slider. It draws the activation curve with its slope, the loss value and the gradient with respect to the logit as the logit moves from confidently right to confidently wrong, so the squared-error gradient can be seen shrinking while the cross-entropy gradient keeps its size.

Four ideas for today Overview

Bend
An activation bends each unit's sum; without it layers collapse (Day 31).
Slope
Backpropagation multiplies slopes; flat regions pass almost nothing back.
Logit
The raw output, any real number. A probability is made from it.
Loss
Chosen from the label's structure; it sets the learning signal for each mistake.

Each activation with its slope; shaded where almost nothing passes back Illustration

Largest slope 0.25 at z = 0; below 0.05 once |z| > 2.9.

Largest slope 1.00 at z = 0; below 0.05 once |z| > 2.2.

Slope exactly 0 for every negative input, exactly 1 for every positive one.

Slope peaks at 1.13 near z = 1.4, slightly above 1; it crosses 0 at the dip (thin band) and fades below z = -2.4.

activation f(z)
slope f′(z), what backpropagation multiplies by (dashed)
slope below 0.05: almost no signal passes back

Computed from the formulas on 401 points in [−5, 5] (GELU uses the tanh approximation, as in the lab code). ReLU's slope at exactly 0 is set to 0 by convention.

The dashed slope is what backpropagation multiplies by (Day 33). Sigmoid's slope never exceeds 0.25; ReLU's is exactly 0 or 1.

Slope is what backpropagation multiplies Table

Activation	Output range	Slope at z = 0	Slope at z = -4 and 4	Typical place
Sigmoid	(0, 1)	0.25 (its maximum)	0.0177 and 0.0177	Binary output unit
Tanh	(-1, 1)	1	0.0013 and 0.0013	Recurrent cells (Day 37)
ReLU	[0, infinity)	undefined; 0 by convention	0 and 1	Default hidden unit
GELU	about [-0.17, infinity)	0.5	about 0 and about 1	Transformer blocks (Day 39)

Values computed from the formulas; the sigmoid, tanh and ReLU slopes match the application printout. GELU dips to its minimum of about -0.17 near z = -0.75, then tracks ReLU; the slope panels above draw all four curves.

The label decides the output layer and the loss Table

Task	Output layer	Loss	Gradient wrt logit
Binary (fraud yes or no)	1 logit, sigmoid	Binary cross-entropy	p - y
Exactly one of K classes	K logits, softmax	Categorical cross-entropy	p_k - 1 if k is the true class, else p_k
Several of K labels	K logits, K sigmoids	Binary cross-entropy per label	p_k - y_k
A number (units sold, EUR)	1 linear unit	MSE, MAE or Huber	MSE: 2 (prediction - y); MAE: sign of the error

A positive case (y = 1) as the logit moves from very wrong to right Illustration

At z = -6: cross-entropy 6.00, squared error 0.995. Squared error can never exceed 1, however wrong.

Squared error's gradient peaks at 0.296 (z = -0.70, p = 0.33) and then dies as the mistake grows.

How many times more signal cross-entropy sends: 1 / (2p(1 − p)). About 203x at z = -6, and it doubles roughly every 0.7 of logit further left.

cross-entropy, gradient p − y
squared error on the sigmoid, gradient 2(p − y)·p(1 − p)
dashed: the application block's example, z = -6

Computed exactly from the two loss formulas for y = 1 on 241 logits in [−8, 4]; no data involved.

Squared error through a sigmoid caps the loss at 1 and kills the gradient for confident mistakes; cross-entropy keeps sending p - y.

Derivative primer: the slope of $z^2$ at 3 is $2z = 6$ ; for $f(g(z))$ the slope is $f'(g(z))\,g'(z)$ (chain rule; do not drop the inner factor).
Sigmoid and tanh flatten in both tails (Saturation); ReLU has slope 0 for negative inputs; GELU (Transformers, Day 39) is a smooth ReLU.
Choose the output layer from the label, then the matching loss: Cross-entropy loss is $-\log p_{\text{true}}$ .
Squared error on a sigmoid has gradient $2(p - y)\,p(1 - p)$ : it vanishes when the model is confidently wrong. Cross-entropy leaves $p - y$ .

Assumptions

Output units and loss follow the label's structure (one of K, several of K, a number); a probability output is trustworthy only after a calibration check (Day 13).
Softmax classes are mutually exclusive and exhaustive; if "none of these" can happen it needs its own class.
The loss is what the optimizer minimizes; accuracy or EUR cost is what you report (Day 16).

Application and workplace decision · 10 min

One booking with $y = 1$ (a no-show) and logit $z = -6$ : the network says "will show up" with $p \approx 0.0025$ . Before running, predict:

which loss value is larger, cross-entropy or squared error;
which gradient with respect to $z$ is larger, and by roughly what factor.

The block then runs a finite-difference slope check, the two gradients, naive versus stable softmax on logits near 1000, and the slopes of three activations at $z = -4, 0, 4$ .

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# Derivative primer: slope of f(z) = z**2 at z = 3 is 2z = 6; a finite difference agrees
f = lambda z: z ** 2
print("slope of z^2 at 3: analytic 6, finite difference", round((f(3 + 1e-6) - f(3 - 1e-6)) / 2e-6, 4))

# One booking that did no-show (y = 1); the network is confidently wrong: logit z = -6
y, z = 1.0, -6.0
p = sigmoid(z)
bce, mse = -np.log(p), (p - y) ** 2
g_bce = p - y                         # dBCE/dz = p - y
g_mse = 2 * (p - y) * p * (1 - p)     # dMSE/dz = dL/dp * dp/dz (chain rule)
print(f"p = {p:.4f} | loss: BCE {bce:.3f}, MSE {mse:.3f}")
print(f"gradient wrt logit: BCE {g_bce:.4f}, MSE {g_mse:.5f} -> {abs(g_bce / g_mse):.0f}x smaller")

# Logits vs probabilities: softmax overflows on large logits, log-softmax with the max trick does not
logits = np.array([1000.0, 1001.0, 999.0])
with np.errstate(over="ignore", invalid="ignore"):
    naive = np.exp(logits) / np.exp(logits).sum()
s = logits - logits.max()
log_softmax = s - np.log(np.exp(s).sum())
print("naive softmax:", naive)
print("stable log-softmax:", np.round(log_softmax, 4), "| probabilities sum", np.exp(log_softmax).sum().round(4))
for name, d in [("sigmoid", lambda a: sigmoid(a) * (1 - sigmoid(a))),
                ("tanh", lambda a: 1 - np.tanh(a) ** 2),
                ("relu", lambda a: (a > 0) * 1.0)]:
    print(f"{name:>7} slope at z = -4, 0, 4:", np.round(d(np.array([-4.0, 0.0, 4.0])), 4))

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 27 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The loss derived: from maximum likelihood to p - y6 minCross-entropy derived from the likelihood of the labels, its gradient derived term by term for two classes and for K, and both worked by hand.

Cross-entropy is not a choice; it is maximum likelihood Sequence

The model
For one example, softmax (or a sigmoid) turns the logits into P(y | x) = p_y, a probability for each class
Likelihood
For N independent examples, the probability the model gives the observed labels is the product of the p_true values
Log and flip
Maximizing a product is maximizing the sum of logs; flip the sign and average to get a quantity to minimize
Name it
That quantity, mean -log p_true, is exactly the cross-entropy loss
Consequence
Training on cross-entropy is fitting the label distribution by maximum likelihood; MSE plays the same role for Gaussian noise (table below)

The derivation is three lines; everything else in this lesson follows from differentiating it.

Binary case by hand: a churn model scores a customer who stayed (y = 0) at logit 1 Sequence

Sigmoid slope
p = 1 / (1 + e^-z). Differentiating gives e^-z / (1 + e^-z)^2, which factors as p (1 - p)
Loss in p
L = -[y ln p + (1 - y) ln(1 - p)], so dL/dp = -y / p + (1 - y) / (1 - p)
Chain rule
dL/dz = dL/dp x p (1 - p) = -y (1 - p) + (1 - y) p = p - y: the slope factor cancels exactly
Numbers
z = 1 gives p = 0.731. Loss -ln(1 - 0.731) = 1.313 nats; dL/dp = 1 / 0.269 = 3.718; slope 0.731 x 0.269 = 0.197
Gradient
3.718 x 0.197 = 0.731 = p - y, and a central difference agrees to 1e-9. Squared error would send only 0.287

The gradient is positive, so lowering the logit lowers the loss, as it should for a customer who stayed. Illustrative churn numbers; every value recomputed in numpy.

Softmax plus cross-entropy, differentiated Formula

L = -z_y + \log \sum_j e^{z_j}, \qquad \frac{\partial L}{\partial z_k} = p_k - \mathbb{1}[k = y]

$z_y$: logit of the true class
$p_k$: softmax probability of class k
$\mathbb{1}[k = y]$: 1 for the true class, 0 otherwise: probabilities minus one-hot target

The sigmoid case is K = 2. This is why the output-layer gradient on Day 33 is so simple.

Deriving the gradient: probabilities minus one-hot, step by step Sequence

Loss in logits
L = -log p_y with p_y = e^{z_y} / sum_j e^{z_j}, so L = -z_y + log sum_j e^{z_j}
First term
d(-z_y)/dz_k = -1 if k is the true class, else 0: the one-hot target
Second term
d/dz_k of log sum_j e^{z_j} = e^{z_k} / sum_j e^{z_j} = p_k: the softmax itself reappears
Add them
dL/dz_k = p_k - 1[k = y]: predicted probability minus target, for every class at once
Sanity checks
The K entries sum to (sum of p_k) - 1 = 0, and the K = 2 case collapses to p - y, the binary gradient

No approximation anywhere; the clean p - y form is why the loss is paired with the softmax rather than applied after it.

Worked by hand: a 3-class return-reason classifier (e-commerce, illustrative logits) Sequence

Logits
z = (1, -1, 0) for damaged / wrong size / changed mind; e^z = (2.718, 0.368, 1.000), sum 4.086
Softmax
p = (0.665, 0.090, 0.245): divide each e^z by the sum
Loss
True reason is damaged, the first class: L = -ln 0.665 = 0.41 nats
Gradient
p minus one-hot (1, 0, 0) = (-0.335, 0.090, 0.245); the three entries sum to 0
Read it
The true class's logit is pushed up by 0.335; each wrong class is pushed down by exactly the probability it claimed

Arithmetic verified in numpy. Redo it on paper with any three logits; only the true class's entry goes negative.

Stable log-softmax for logits (1000, 1001, 999) Sequence

Naive
exp(1000) overflows to infinity, so the ratio is nan
Subtract the max
Logits minus 1001 give (-1, 0, -2); softmax is unchanged by a shift
Log-sum-exp
log(e^-1 + e^0 + e^-2) = 0.4076
Log-probabilities
(-1.4076, -0.4076, -2.4076); they exponentiate to a valid distribution

Losses that take logits do exactly this internally, which is why the model should not apply softmax first.

Why cross-entropy and not squared error4 minThe chain rule shows where the extra factor comes from, and a training run shows what it does to a confident mistake.

Why squared error goes quiet Formula

\frac{\partial L}{\partial z} = \underbrace{2(p - y)}_{\partial L/\partial p} \cdot \underbrace{p(1 - p)}_{\partial p/\partial z}

$p = \sigma(z)$: the sigmoid probability
$p(1 - p)$: sigmoid slope: at most 0.25, near 0 when p is near 0 or 1
$2(p - y)$: the error term; largest when confidently wrong, but multiplied by a tiny slope

Cross-entropy's derivative wrt p is -1/p for y = 1, which cancels the slope exactly and leaves p - y.

Cross-entropy -log p for the probability given to the true class Chart

p = 0.90.105 nats
p = 0.5 (a coin flip)0.693 nats
p = 0.12.303 nats
p = 0.01confident mistakes cost the most4.605 nats

A uniform guess over K classes scores ln K at initialization; check that number before training (Day 34). Cross-entropy is a strictly proper scoring rule: its expected value is lowest when the reported probability is the true one.

Same unit, same data, same bad start, two losses Illustration

Both start at log loss 23.2. After 300 steps: cross-entropy 0.200, squared error 23.2.

The weight w: cross-entropy flips it to 4.7 (accuracy 92%); squared error leaves it at -29.9 (accuracy 9%).

trained with cross-entropy
trained with squared error on the sigmoid

Computed: 400 seeded points, x standard normal, label 1 when x plus noise (sd 0.3) is positive; one sigmoid unit z = w x + b, start w = −30, b = 0, learning rate 0.5, full-batch steps. Log loss is reported for both runs so they share a scale.

Only the gradient formula differs. Cross-entropy climbs out of a confidently wrong start; squared error barely moves the weight.

The trouble is the pairing, not squared error itself Map

Quiet when confidently wrong

Through a sigmoid the gradient is 2(p - y) p(1 - p), near 0 in both tails.

The right pairing

Gradient p - y. Strictly proper: minimized by reporting the true probability.

Still the right loss

Linear output, gradient 2(prediction - y): maximum likelihood for Gaussian noise.

Undefined

Cross-entropy needs a probability per class; a sales figure has none.

Squared error is fine for a real-valued target with a linear output unit; it fails only behind a squashing output. Cross-entropy outputs become usable probabilities only after a calibration check (Day 13).

Activation derivatives, computed, and dead units4 minThe three derivatives in closed form, which activation to use where, and how ReLU units die.

The three derivatives backpropagation will multiply by (Day 33) Formula

\sigma'(z) = \sigma(z)\,(1 - \sigma(z)), \quad \tanh'(z) = 1 - \tanh^2(z), \quad \operatorname{ReLU}'(z) = \mathbb{1}[z > 0]

$\sigma'$: differentiate (1 + e^{-z})^{-1} and factor; the slope peaks at 0.25 at z = 0 and is 0.0177 by |z| = 4
$\tanh'$: peaks at 1 at z = 0; down to 0.0013 by |z| = 4, the fastest-saturating of the three
$\operatorname{ReLU}'$: exactly 0 or 1; the kink at 0 is assigned slope 0 by convention

These closed forms are what the application block checks against finite differences at z = -4, 0, 4.

Choosing a hidden activation Comparison

Criterion	Sigmoid or tanh	ReLU	GELU
Slope behaviour	At most 0.25 (sigmoid) or 1 (tanh); near 0 in the tails	Exactly 1 for positive inputs, 0 for negative	Smooth; nonzero slope on both sides of 0
Main risk	Saturation and vanishing gradients in deep stacks	Dead units: negative for every row, never updated	Slightly more compute than ReLU
Where you meet it	Output units, gates of recurrent cells	Default hidden unit in MLPs and CNNs	Transformer feed-forward blocks

Dead ReLU units Overview

Symptom
A unit outputs 0 for every training row, so its gradient is 0 and it never changes.
Common cause
A large learning-rate step pushes the bias far negative.
Monitor
Share of units that are zero on every row of a batch, per layer.
Remedy
Lower the rate; leaky variants keep a small slope for negative inputs.

Choosing the loss, and what the library expects3 minA decision flow from label to loss, and the input each common loss class expects.

From the label to the loss Diagram

Label is a number?
- yes: to Outliers or a count?
no
Exactly one class?
- yes: to Softmax cross-entropy
- no: to Sigmoid BCE per label
Outliers or a count?
- no: to MSE
- yes: to Huber, MAE or Poisson
Softmax cross-entropyK logits
- then ask: to Rare positive class?
Sigmoid BCE per labelK logits, multi-label
- then ask: to Rare positive class?
MSEGaussian noise
Rare positive class?
- yes: to Class weights
Huber, MAE or Poissonrobust or count
Class weightsor pos_weight; then check calibration

Weighting changes the probabilities the model learns; re-check calibration and thresholds afterwards (Days 13 and 16).

What each loss expects as input Table

Loss	Model output it expects	Notes
PyTorch CrossEntropyLoss	Unnormalized logits, one per class	Options: class weight, ignore_index, label_smoothing
PyTorch BCEWithLogitsLoss	Logits, one per label	Sigmoid inside, log-sum-exp stable; pos_weight for rare positives
PyTorch BCELoss	Probabilities	Needs a separate sigmoid; less stable at confident outputs
PyTorch NLLLoss	Log-probabilities	Pair with LogSoftmax; CrossEntropyLoss is the two combined
scikit-learn log_loss	Probabilities (predict_proba)	A metric for reporting, not a training loss

From the PyTorch CrossEntropyLoss and BCEWithLogitsLoss documentation (2.14 pages) and the lesson code. Feeding probabilities where logits are expected is the double-softmax bug.

The double-softmax bug2 minWhat applying softmax twice does to the best probability and the loss.

Softmax applied twice: the best possible probability and the loss floor Table

Classes K	Best probability e / (e + K - 1)	Loss floor	Uniform loss ln K
2	0.731	0.313	0.693
5	0.405	0.905	1.609
10	0.232	1.461	2.303

A model that ends in softmax, fed to a loss that applies softmax again, can never be confident. Accuracy can still look fine.

Softmax twice puts a floor under the loss; logits straight in remove it Diagram

Logits zthe model's raw output
- to Softmax in the model
Logits zno final softmax
- logits go in: to Loss: log-softmax
- when needed: to Softmax at inference
Softmax in the modeloutputs squeezed into [0, 1]
- to Loss: log-softmax
Softmax at inferenceonly where a person or threshold needs it
Loss: log-softmaxsquashes those values again
- to Loss floor above 0
Loss: log-softmaxapplied once, stable
- to Loss can reach 0
Loss floor above 0probabilities never approach 1
Loss can reach 0full range of probabilities

Top row: the bug. Bottom: the fix. Tell-tale of the top row: a training loss that stalls above zero and probabilities that never approach 1, while accuracy looks fine because softmax keeps the ranking.

Read the loss at step 02 minWhat the first printed loss should be, and which bug each deviation points to.

The loss at step 0 is a free bug detector Sequence

Expect
All logits near 0 give uniform probabilities, so the loss is about ln K (ln 2 = 0.693 for a binary output).
Far above ln K
Logits start large: bad init scale, unscaled inputs, or a bias set wrongly.
Stuck at a floor
A probability-squashing layer applied twice; the loss cannot reach 0.
Rare class
Setting the output bias to log(prior / (1 - prior)) starts at the base rate instead of 0.5.

Practitioner checks in the spirit of Karpathy's recipe (2019); the bias trick is the logit of the class prior.

Losses as noise assumptionsadvanced3 minEach loss is maximum likelihood for a noise model, which is how to pick one for counts or outliers.

Each loss is a noise assumption Table

Loss	Maximum likelihood for	Use when
Cross-entropy	Bernoulli or categorical label	Classification
Squared error (MSE)	Gaussian noise, constant variance	Symmetric, light-tailed errors
Absolute error (MAE)	Laplace noise; predicts the median	Outliers in the target
Huber	Gaussian-like near 0, Laplace-like tails	Mostly Gaussian with occasional outliers
Poisson	Count data	Units sold, claims per month, calls per hour
Quantile (pinball)	A chosen quantile	Forecast intervals (Day 26)

Weights, smoothing and focal lossadvanced3 minThe common modifications of cross-entropy, what each buys and what it costs in calibration.

Knobs on the loss, and what each one costs Overview

Class weights
Up-weight a rare class (weight, pos_weight). Probabilities shift: recalibrate before thresholding.
Label smoothing
Mixes the target with a uniform one; can improve calibration but weakens distillation (Muller et al., 2019).
Focal loss
Down-weights easy examples so many easy negatives do not dominate (Lin et al., 2017).
Overconfidence
Modern networks trained on cross-entropy can be overconfident; check calibration (Guo et al., 2017).

Change the loss only when the rare class needs it, then re-earn the threshold Diagram

Rare-class metric poor?
- yes: to Weight, smooth or focal
no
Keep plain cross-entropyweights and resampling are not free
- still report: to Report calibration too
Weight, smooth or focalone knob at a time
Recalibrate (Day 13)a predicted 0.8 now means something else
Re-pick the thresholdfrom costs on validation data (Day 16)
ship with
Report calibration toonext to the metric that matters

A staff-level answer: try resampling or weighting only when the metric that matters is poor for the rare class. Every modification changes what a predicted probability means, so thresholds chosen before the change are stale.

Memory anchor

The model speaks logits, the loss reads logits, and only people read probabilities.

Useful for

Choosing the last layer and loss for a new target (binary, multi-class, multi-label, regression) before any training run.
Reviewing training code for double activations, probabilities fed to a logit loss, and unstable manual softmax.
Explaining to stakeholders why a score is not yet a probability and what must be checked before it is treated as one.

Not suitable for

Choosing a decision threshold or an action policy - the loss trains the scores, while thresholds come from costs (Day 16).
Deciding whether the scores are well calibrated - cross-entropy is a proper scoring rule, but calibration needs its own check on held-out data (Day 13).

Traps

Softmax in the model, logit-expecting loss on top

The model ends in softmax or sigmoid and the loss applies it again. Accuracy still looks fine because the ranking of scores is preserved, but the loss has a floor, gradients are weak, and the probabilities are compressed.

How to avoid it

Return logits from the model, let the loss apply log-softmax or sigmoid internally, and check the loss at initialization against $\ln K$ .

Treating a softmax output as calibrated confidence

A network trained well on cross-entropy can still be overconfident on new data, so "0.97" does not mean 97% of such cases are right. Thresholds and expected-cost decisions then use inflated confidence.

How to avoid it

Measure calibration on held-out data and recalibrate (Day 13) before using the scores as probabilities.

Senior reasoning

A senior chooses the output layer from the label structure and the decision, then lets the loss see logits. They check the loss at initialization (about $\ln K$ for K balanced classes, $\ln 2$ for a zero logit), look at the maximum probability the model can produce, and keep reported metrics separate from the training loss. When a score is used as a probability they ask for a calibration check first. They would rather lose an hour to a gradient and loss check than debug a plateau that is a double softmax.

Interview question

"Why do we train classifiers with cross-entropy and not mean squared error?" How would you answer?

Outline your answer first.

Recall cards scheduled after completion

Which output layer and loss do you use for binary, one-of-K, multi-label and numeric targets?
Why does sigmoid plus squared error train poorly when the network is confidently wrong?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Dive into Deep Learning — Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola, 2023core
Softmax regression and cross-entropy, the log-sum-exp stability argument, and the activation functions used in multilayer perceptrons. (Chapter 4 (softmax regression) and Chapter 5.1 (multilayer perceptrons))
CrossEntropyLoss (PyTorch documentation) — PyTorchcore
Documents that the loss expects unnormalized logits per class, which is why applying softmax in the model is a bug.
BCEWithLogitsLoss (PyTorch documentation) — PyTorchoptional
Combines sigmoid and binary cross-entropy in one operation that is more stable than a separate sigmoid followed by BCELoss.
Understanding the difficulty of training deep feedforward neural networks — Xavier Glorot, Yoshua Bengio, 2010optional
Shows that sigmoid hidden units drive the top hidden layer into saturation under random initialization and slow learning, motivating other activations.
Gaussian Error Linear Units (GELUs) — Dan Hendrycks, Kevin Gimpel, 2016optional
Defines GELU as the input times the standard normal cumulative distribution function, the smooth activation used in Transformer models.
On Calibration of Modern Neural Networks — Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger, 2017optional
Reports that modern neural networks trained with cross-entropy can be poorly calibrated, the reason to check calibration before reading outputs as probabilities.
When Does Label Smoothing Help? — Rafael Muller, Simon Kornblith, Geoffrey Hinton, 2019optional
Label smoothing can improve calibration but makes distillation from the smoothed model less effective.
Focal Loss for Dense Object Detection — Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollar, 2017optional
Focal loss reshapes cross-entropy to down-weight well-classified examples under extreme class imbalance.
A Recipe for Training Neural Networks — Andrej Karpathy, 2019optional
Practitioner checklist including checking the loss at initialization and initializing the output bias from the class prior; cited as experience, not as a study.
Deep Learning — Ian Goodfellow, Yoshua Bengio, Aaron Courville, 2016optional
Output units and the matching cost functions (maximum likelihood), and hidden-unit choices including ReLU and its generalizations. (Chapter 6.2 (cost functions, output units) and 6.3 (hidden units))

Optional sections

Deep dive · 10 min

Softmax plus cross-entropy gradient. For one example with true class $y$ , $L = -\log p_y = -z_y + \log \sum_j e^{z_j}$ . Differentiating with respect to a logit $z_k$ gives $-\mathbb{1}[k = y] + e^{z_k} / \sum_j e^{z_j}$ , which is $p_k - \mathbb{1}[k = y]$ : predicted probability minus one-hot target. The sigmoid case is the K = 2 special case. This is why the output-layer gradient on Day 33 is so simple.

Why these losses. Minimizing cross-entropy is maximum likelihood for a categorical (or Bernoulli) label; minimizing squared error is maximum likelihood for a Gaussian error with constant variance. The loss encodes an assumption about the noise, which is why a count or a heavy-tailed target may call for a Poisson or quantile loss instead (quantile forecasts, Day 26).

GELU, exactly. $\text{GELU}(z) = z\,\Phi(z)$ where $\Phi$ is the standard normal cumulative distribution function. For large positive $z$ it is close to $z$ , for large negative $z$ close to 0, and it dips slightly below zero near $z = -1$ (the lab prints -0.159 there). Unlike ReLU it has a nonzero slope on both sides of 0.

Dead units. A ReLU unit whose pre-activation is negative for every training example has zero gradient and never changes again. A large learning-rate step is a common cause (monitor the share of always-zero units); leaky variants keep a small slope for negative inputs.

Hands-on lab: Same unit, same start, two losses; and a softmax applied twice · 15 min

Run the block. Before reading the output, predict which loss recovers from a start of $w = -30$ (a sigmoid unit that is confidently wrong on most examples) in 300 steps, and what accuracy the other one ends at.

Then answer: (1) Why does the MSE run stay near its starting weight even though its gradient points the right way? (2) What probability and loss floor does a double softmax impose for three classes, and how would that show up in a training log? (3) Where does GELU differ visibly from ReLU in the printed grid?

Extension: change the starting weight to $-6$ and re-run. Saturation matters less when the start is mildly wrong; note what that says about checking the loss at initialization.

import math
import numpy as np

rng = np.random.default_rng(3)
sigmoid = lambda z: 1 / (1 + np.exp(-z))

# 1) One sigmoid unit, started confidently wrong (w = -30), trained with each loss on the same synthetic data
n = 400
x = rng.normal(size=n)
y = (x + 0.3 * rng.normal(size=n) > 0).astype(float)

def fit(loss, steps=300, lr=0.5, w=-30.0):
    b = 0.0
    for _ in range(steps):
        p = sigmoid(w * x + b)
        dz = (p - y) if loss == "bce" else 2 * (p - y) * p * (1 - p)
        w -= lr * np.mean(dz * x)
        b -= lr * np.mean(dz)
    p = np.clip(sigmoid(w * x + b), 1e-12, 1 - 1e-12)
    return w, ((p > 0.5) == y).mean(), -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))

for loss in ("bce", "mse"):
    w, acc, ll = fit(loss)
    print(f"{loss.upper()} after 300 steps: w = {w:7.2f}, accuracy {acc:.3f}, log loss {ll:.3f}")

# 2) Softmax applied twice (a model that ends in softmax, fed to a loss that applies softmax again)
best_first = np.array([1.0, 0.0, 0.0])           # the most confident output a softmax layer can produce
second = np.exp(best_first) / np.exp(best_first).sum()
print(f"double softmax, 3 classes: best possible probability {second[0]:.4f}, loss floor {-np.log(second[0]):.4f}")

# 3) GELU, z * Phi(z), next to ReLU on a small grid
gelu = lambda a: a * 0.5 * (1 + np.vectorize(math.erf)(a / math.sqrt(2)))
grid = np.array([-3.0, -1.0, -0.5, 0.0, 0.5, 1.0, 3.0])
print("z      ", grid)
print("ReLU   ", np.maximum(0, grid))
print("GELU   ", np.round(gelu(grid), 3))

# 4) Cross-entropy of a 3-class prediction: loss is -log of the probability given to the true class
logits = np.array([2.0, 0.5, -1.0])
s = logits - logits.max()
logp = s - np.log(np.exp(s).sum())
print("probabilities", np.round(np.exp(logp), 3), "| loss if class 0 is true", round(-logp[0], 3), "| if class 2", round(-logp[2], 3))

Interview practice · 10 min

A teammate's PyTorch defect classifier (five defect types from factory sensor features) has a Softmax layer at the end and uses CrossEntropyLoss. Training "works", but the loss never gets very small. Review the code in two minutes: what do you say?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-18 · depends on numpy