Core day 35 of 60 · Week 7 · Part II · evaluation depth

Deep Learning Lab

A 64-32-10 network and a logistic regression, trained on the same digits rows, have accuracies that differ in the third decimal. What can you claim, what can you not claim, and what evidence would you need before telling a team to ship the network?

By the end: From one executed numpy network and a classical baseline on the same split and metric, you write a result statement that quotes only printed numbers, says what the comparison supports and what it does not, and names the next check that could overturn it.

At a glance

Same rows, same metric, comparable tuning effort and several seeds or splits; then write the claim only as strong as the evidence.

Fix three things
The rows used to fit, the rows used to score, and the metric.
Match the effort
A hand-tuned network against a default baseline compares effort, not models.
One run is a sample
Read a gap against its standard error and its spread over seeds and splits.
Count disagreements
Rows where exactly one model is right decide a paired comparison.
Claim carefully
Say what the numbers support, what they do not, and the next check.

Recall · 5 min

Reviews due from Days 1, 9, 13 and 31-34 appear first. If none are due: list the four checks you run on a new network before comparing it with anything (shapes and parameter count, loss at initialization, gradient check, overfit a tiny batch), and state what the standard error of an accuracy depends on.

Open due reviews

Concept · 10 min

A neural-network project ends in one sentence: "the network is better than the baseline." Today is about earning that sentence, or not saying it. Same rows, same metric, comparable tuning effort, and a gap judged against how far it moves when you re-split or re-seed (Seed spread).

A network beats a baseline by a wide margin on curved data and ties it on linear data, and one seed is only a sample

Predict first

How many validation rows separate 96.4% and 96.7% accuracy on 360 rows?

Text description of the figure

A small two-input network lab on synthetic two-moons or two-blobs data (not digits), with hidden-unit, learning-rate, epoch and seed controls. It shows the layer shapes and parameter count, the training and validation loss curves with the logistic regression and class-prior baselines, a table of validation log loss and accuracy for the three models on the same split, a decision-region plot, and a five-seed table.

The comparison protocol, in order Sequence

Freeze the split
Stratified and seeded: train, validation, test. The test part stays sealed (Days 9 and 15).
Preprocess on training rows
Digits need only a fixed 1/16 rescale, which has nothing to fit.
Strongest simple baseline
Regularized logistic regression, or gradient boosting for tabular data.
Verify the network
Shapes, initial loss near ln 10 = 2.30, gradient check, overfit a tiny batch (Days 31-34).
Same metrics
Accuracy counts decisions; log loss scores probabilities, which training minimized.
Quantify noise
Standard error, paired disagreements, several splits and seeds.
Write the claim
At the strength the evidence supports, with the next check that could overturn it.

What must be identical before any comparison means something Overview

Rows to fit
Both models train on exactly the same rows.
Rows to score
Both models are scored on exactly the same held-out rows.
Metric
The same metric, computed the same way, for both.
Tuning effort
A comparable search for both, chosen without touching the test part.

Same split, same metric: when the network earns its keep, and when it does not Illustration

Validation log loss 0.386, accuracy 76.0%

Validation log loss 0.194, accuracy 91.7%

Validation log loss 0.095, accuracy 95.8%

Validation log loss 0.114, accuracy 93.8%

Computed with the nn-lab code at its defaults (8 tanh units, Adam rate 0.02, 150 epochs, seed 1): 144 training and 96 validation points per dataset; dots are the validation points both models are scored on. A different seed moves the network's numbers (next figure).

On curved data the network's bent boundary beats the straight line by a wide margin. On data a line already separates, the network adds parameters and nothing else.

One seed is one sample Illustration

network, one dot per initialization seed (6)
logistic regression, same rows (deterministic)

Computed with the nn-lab code at its defaults (2-8-1 tanh, Adam rate 0.02, 150 epochs); only the network's initialization seed changes. The split is fixed, so split-to-split noise comes on top of this spread.

moons: network 0.176–0.201 (sd 0.010) against 0.386, better on 6 of 6 seeds; blobs: network 0.108–0.118 (sd 0.003) against 0.095, better on 0 of 6 seeds. Moons: the gap is many times the seed spread. Blobs: a straight line is already the right shape, and the extra flexibility costs a little on every seed.

Only the initialization seed changes between dots. A gap counts when it is large against the spread of the dots, and split-to-split noise comes on top.

Today's 360 validation rows, paired: only 3 rows carry evidence Grid

Logistic regression →
Network ↓	Right	Wrong
Right	346	1
Wrong	2	11

Both right (346) says nothing about which model is better; both wrong (11) are hard digits to inspect before adding capacity. The off-diagonal cells are the discordant rows: 1 for the network, 2 for logistic regression. Split 1 to 2, an exact two-sided binomial (McNemar) test gives p = 1.0: no difference shown.

How strong a claim each kind of evidence supports, from anecdote to decision-grade Sequence

One split, one seed
Today's core block: an anecdote. It can show a tie or a hint, never a ranking
Several seeds
Initialization noise becomes visible; a gap smaller than the seed spread is not a finding
Splits plus a tuned baseline
The lab: data-sampling noise and tuning effort are now on both sides of the comparison
One look at the sealed test
Unbiased for the single configuration chosen beforehand; looking twice turns it into validation
Live comparison
Shadow mode or an A/B test on real traffic: decision-grade, because it measures the deployment itself

Each rung removes one source of doubt that the rung below leaves open. State the rung your evidence reached next to the claim.

Follow the seven-step protocol in the pictures: freeze the split, fit preprocessing on training rows, build the strongest simple Baseline, verify the network, compare on the same metrics, quantify noise, write the claim.
An accuracy $p$ on $n$ rows has standard error $\sqrt{p(1-p)/n}$ ; for two models on the same rows, the rows where exactly one is right (a Paired comparison) decide.
Data sampling, initialization and hyperparameters all move such comparisons (Bouthillier et al., 2021).
On tabular business data, tree ensembles are hard for a small network to beat (Grinsztajn et al., 2022; Shwartz-Ziv and Armon, 2022): a prior, not a law. Digits are images, where a CNN (Day 36) is the natural neural model.

Assumptions

Validation rows stand in for future data; digits from the same writers and scanner would not predict performance on new ones (Days 18 and 19).
Both models received comparable tuning effort, spent on training rows or on a validation part that is then not reused as the final score.
Differences smaller than seed-to-seed or split-to-split variation are not findings.
A result on 8x8 digits supports a statement about this dataset and these models, not about images or tabular data in general.

Application and workplace decision · 10 min

Commit to a written prediction first:

which model has the higher validation accuracy, and which the lower log loss;
on how many of the 360 validation rows exactly one of the two models will be right.

The block trains a 64-32-10 ReLU network (300 full-batch steps, learning rate fixed in advance) and scikit-learn's LogisticRegression with default settings on the same rows, and prints accuracy, log loss, the disagreements and a binomial standard error. Then write a result statement of at most four sentences: split and sizes, both metrics, what they support, what they do not.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)
X = X / 16.0                                          # fixed rescale; nothing is fitted on any split
dev_X, _, dev_y, _ = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)  # 20% test stays sealed
Xtr, Xva, ytr, yva = train_test_split(dev_X, dev_y, test_size=0.25, stratify=dev_y, random_state=0)

def softmax(Z):
    E = np.exp(Z - Z.max(1, keepdims=True))
    return E / E.sum(1, keepdims=True)

r = np.random.default_rng(0)
W1, b1 = r.normal(0, np.sqrt(2 / 64), (64, 32)), np.zeros(32)
W2, b2 = r.normal(0, np.sqrt(1 / 32), (32, 10)), np.zeros(10)
T, lr = np.eye(10)[ytr], 0.5
for epoch in range(300):                              # full-batch gradient descent, epochs fixed in advance
    H = np.maximum(0, Xtr @ W1 + b1)
    dZ = (softmax(H @ W2 + b2) - T) / len(Xtr)
    dH = (dZ @ W2.T) * (H > 0)
    W2, b2 = W2 - lr * H.T @ dZ, b2 - lr * dZ.sum(0)
    W1, b1 = W1 - lr * Xtr.T @ dH, b1 - lr * dH.sum(0)

p_mlp = softmax(np.maximum(0, Xva @ W1 + b1) @ W2 + b2)
p_lr = LogisticRegression(max_iter=2000).fit(Xtr, ytr).predict_proba(Xva)
ok_mlp, ok_lr = p_mlp.argmax(1) == yva, p_lr.argmax(1) == yva
n = len(yva)
print(f"split: train {len(ytr)}, validation {n}, parameters in MLP {64*32+32+32*10+10}")
for name, p, ok in [("numpy MLP 64-32-10", p_mlp, ok_mlp), ("logistic regression", p_lr, ok_lr)]:
    print(f"{name:<20} accuracy {ok.mean():.3f}  log loss {log_loss(yva, p, labels=range(10)):.3f}")
print(f"only MLP correct: {np.sum(ok_mlp & ~ok_lr)}, only LR correct: {np.sum(~ok_mlp & ok_lr)}, of {n}")
print(f"binomial s.e. of one accuracy: {np.sqrt(ok_lr.mean() * (1 - ok_lr.mean()) / n):.3f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 24 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

How big is the noise in an accuracy?5 minThe binomial standard error derived from 0/1 indicators and worked on 10 rows, a simulation of it, and what validation size buys.

Standard error of an accuracy Formula

\operatorname{SE}(\hat p) = \sqrt{\frac{\hat p\,(1 - \hat p)}{n}}

$\hat p$: observed accuracy
$n$: number of scored rows

A rough 95% interval is plus or minus 2 standard errors. It covers row-sampling noise only, not seeds or tuning.

The standard error of an accuracy, derived Sequence

Indicators
For row i let c_i = 1 if the model is right and 0 if not. Accuracy is their mean: p-hat = (c_1 + ... + c_n) / n
One row
If rows are independent draws and the true accuracy is p, each c_i is Bernoulli(p): mean p, variance p (1 - p)
The mean
The variance of a mean of n independent terms is one term's variance divided by n: p (1 - p) / n
Plug in
Replace the unknown p with p-hat and take the square root: SE = sqrt(p-hat (1 - p-hat) / n)
By hand
8 right out of 10: sqrt(0.8 x 0.2 / 10) = sqrt(0.016) = 0.126, so a rough 95% band is 0.8 plus or minus 0.25
Limits
It covers row-sampling noise only. Near 0 or 1, or on a handful of rows, use an exact or Wilson interval (Days 45 and 50)

Nothing here is specific to neural networks; it is the standard error of a proportion, and every model comparison inherits it.

How far an accuracy moves when only the rows change Illustration

Spread (sd) of the observed accuracy: 2.1 points on 100 rows, 1.0 points on 400 rows, 0.5 points on 1600 rows. The formula √(p(1−p)/n) gives 2.2, 1.1, 0.5. Four times the rows, half the spread.

Two models, both truly 95%, on 400 rows: the observed gap has sd 1.03 points when they share the rows and 1.52 when scored on separate rows. A gap of 1 point or more appears in 40% (shared) and 56% (separate) of draws, with no real difference.

left: ±2 standard errors from the formula
right: same rows (paired)
right: separate rows

Seeded simulation, no real models: each row is right with probability 0.95. Paired models agree on 96% of rows (both right 93%, each alone right 2%), which is why sharing rows narrows the gap.

Left, one model on fresh validation sets of three sizes. Right, two equally good models; scoring them on the same rows (paired) narrows the gap distribution, which is why disagreements, not two intervals, decide.

Validation accuracy on one split (360 rows) Chart

Network 64-32-10347 of 3600.964
Logistic regression348 of 3600.967

The bars are visually identical because the gap is one row. The standard error of either accuracy is about 0.009.

Standard error of a 0.967 accuracy by validation size Table

Validation rows	Standard error	Rows per 0.01 of accuracy
100	0.018	1
360 (today)	0.009	3.6
1,000	0.006	10
10,000	0.002	100

Quadrupling the rows only halves the standard error.

The paired comparison, step by step5 minWhy discordant rows decide, the paired standard error derived through the covariance on 10 claims, and the exact McNemar test worked on today's three rows.

The standard error of a paired accuracy difference Formula

\widehat{\operatorname{SE}}(\hat p_A - \hat p_B) = \frac{\sqrt{b + c}}{n}

$b, c$: discordant counts: rows only model A gets right, rows only model B gets right
$b - c$: the difference in correct counts; concordant rows cancel out of it exactly
$b + c$: each discordant row shifts the difference by one, so the variance of b - c is about their sum
$n$: scored rows, to convert counts into accuracy

Today: b = 1, c = 2, so the gap's standard error is sqrt(3)/360 = 0.48 points, against an observed gap of 0.28 points - well inside noise. Pairing is what makes this smaller than the standard error of either accuracy alone.

Worked by hand: two claims-triage models scored on the same 10 claims (insurance) Sequence

The rows
Claims 1-6: both models right. Claims 7 and 8: only A right. Claim 9: only B right. Claim 10: both wrong. A scores 8/10, B 7/10
Per-row difference
d_i = (A right) - (B right): 0 on claims 1-6 and 10, +1 on 7 and 8, -1 on 9. The mean of d is 0.1, exactly the accuracy gap
Unpaired, wrong here
sqrt(0.8 x 0.2 / 10 + 0.7 x 0.3 / 10) = sqrt(0.016 + 0.021) = 0.192 treats the two scores as independent samples
The covariance
Shared claims make the indicators covary: Cov = 0.6 - 0.8 x 0.7 = 0.04, so Var(gap) = 0.037 - 2 x 0.04 / 10 = 0.029, SE 0.170
Same result from d
Var(d) = 0.3 - 0.1 squared = 0.29; / 10 = 0.029 again. sqrt(b + c) / n = sqrt(3) / 10 = 0.173 drops the small (b - c)^2 / n term
Test
Exact McNemar on b = 2, c = 1 gives p = 1.0. Ten claims prove nothing; the arithmetic is what carries over to 360 rows or 36,000

Illustrative numbers. Concordant claims contribute zeros to d, which is why only disagreements carry evidence. Every value recomputed in numpy, the p-value with scipy's binomtest.

A paired test on discordant rows (McNemar, exact form) Sequence

Count
b = rows only model A gets right, c = rows only model B gets right
Null hypothesis
Equal accuracy: each discordant row is a fair coin flip
Test
Exact binomial test of b out of b + c at probability 0.5
Today
b = 1, c = 2, so 1 of 3: two-sided p = 1.0

The pairing is why two close accuracies on the same rows need the disagreements, not two separate intervals.

The exact McNemar test on today's 3 discordant rows, worked Table

k = rows won by the network	P(k) if a fair coin decided each	Running total
0 of 3	1/8 = 0.125	0.125
1 of 3 (observed)	3/8 = 0.375	0.500
2 of 3	3/8 = 0.375	0.875
3 of 3	1/8 = 0.125	1.000

Under equal accuracy each discordant row is a coin flip, so k is Binomial(3, 1/2). One-sided tail P(k <= 1) = 0.5; doubling and capping at 1 gives the exact two-sided p = 1.0 (scipy binomtest agrees). Three discordant rows cannot distinguish anything.

Pair log loss per row too: one interval on the difference, not two side by side Diagram

Per-row log lossboth models, the same 360 rows
- unpaired: to Two separate SEs
Row differences dnetwork minus baseline, one per row
- to Mean and SE of d
Two separate SEs0.036 and 0.024: gap SE 0.043
Mean and SE of d-0.028, SE 0.016
One intervalresample rows, 95%: -0.055 to 0.005

Computed from the application block's rows (10,000 bootstrap resamples of rows). Shared rows cancel the difficulty both models face, so the paired standard error is under half the unpaired one. Even the log-loss edge on this split is about 1.8 standard errors: suggestive, not settled.

Seeds, splits and the variance you forgetadvanced4 minSeed noise measured on today's network, the four sources of spread, and why repeated splits need a corrected variance.

Five initialization seeds, same split, same recipe (measured) Table

Seed	Validation accuracy	Validation log loss
0 (the core block)	0.964	0.144
1	0.969	0.131
2	0.958	0.147
3	0.964	0.149
4	0.967	0.136
mean +/- sd	0.964 +/- 0.004	0.141 +/- 0.008

Recorded by rerunning the application block with only the weight-init seed changed. The accuracy moves over a 1.1-point range from the seed alone, so the 0.3-point gap to the baseline on seed 0 is smaller than initialization noise, before split noise is even counted.

Where the spread in a comparison comes from Stack

Data samplingWhich rows land in train and validation; re-split to see it.
Initialization and data orderSeed of the weights and of the batch shuffling; re-seed to see it.
Hyperparameter choiceHow hard each model was tuned, and on which rows.
Hardware and libraryNon-deterministic kernels and versions; same seed, slightly different numbers.

Bouthillier et al. (2021) show the first three can change benchmark conclusions; vary them on purpose before ranking models.

Vary seeds and splits together, or the study hides half the noise Map

Seed-only study

Shows initialization noise; hides split-to-split noise.

Seeds and splits together

The spread a gap must beat.

One split, one seed

Today's core block: an anecdote, never a ranking.

Split-only study

Shows data-sampling noise; hides initialization noise.

Fix the number of repeats before running, so the count is not chosen after seeing which way the gap leans. Repeated splits share rows, so their variance needs the correction below.

Repeated splits overlap, so correct the variance Formula

\hat\sigma^2_{\text{corr}} = \Big(\frac{1}{J} + \frac{n_{\text{test}}}{n_{\text{train}}}\Big)\, \hat\sigma^2

$\hat\sigma^2$: variance of the per-split score differences
$J$: number of repeated splits
$n_{\text{test}} / n_{\text{train}}$: penalty for training sets that share most of their rows

The corrected resampled t-test (Nadeau and Bengio, 2003), as used in scikit-learn's model-comparison example: without the second term, more splits look like more evidence than they are.

Accuracy and log loss can disagree2 minWhat each metric rewards and when to lead with which.

Why log loss moved while accuracy did not Comparison

Criterion	Accuracy	Log loss
What it scores	Only the argmax decision for each row	The probability given to the true class
Sensitive to	Rows near the decision boundary	Confidence on every row, right or wrong
Use it when	Counting correct readings is the decision	Thresholds, cost weighting or calibrated scores drive the decision

Regularization strength can change probabilities on many rows without changing many decisions.

Fair comparisons and the sealed test part3 minTuning effort, identical rows and one look at the test part.

A fair comparison versus a flattering one Comparison

Criterion	Flattering	Fair
Tuning	Network tuned by hand for days; baseline at defaults	Comparable search for both, on training or validation rows only
Rows	Different splits or preprocessing fitted on all rows	Identical rows, preprocessing fitted on training rows
Noise	One seed, best run reported	Several seeds and splits, spread reported
Test part	Looked at repeatedly to pick the winner	Opened once, for the single chosen configuration

What the sealed test part is for Diagram

Develop on train + valall tuning and choices
One config chosen?
- no, peeking: to Test used to choose
yes
Score test onceunbiased for that choice
- to Report and decide
Test used to choosenow a validation set
Report and decide

Looking at the test part to pick among candidates makes the final number optimistic by an unknown amount.

From a gap to a decision3 minFour questions a gap must survive, and the cases where a network is worth it even at a tie.

From a gap to a decision Diagram

Gap beyond paired noise?
- no: to Report a tie
yes
Survives splits + seeds?
- yes: to Beats a tuned baseline?
no
Report a tiekeep the simpler model
Beats a tuned baseline?
- no: back to Report a tie
yes
Pays its running cost?
yes
Sealed test, one lookthen decide

Paired noise means the disagreements and the per-row loss differences; running cost means latency, serving and monitoring. Each "no" ends in the same honest sentence. Only a gap that survives all four questions earns the sealed test part.

When an expert picks the network despite a tie Overview

Shared representation
Several related targets or an embedding reused downstream (Days 38-39).
Raw inputs
Pixels, text or sequences that a fixed feature table loses (Days 36-37).
Much more data coming
A curve still rising with data size, measured, not assumed.
The cost case still holds
The serving, monitoring and retraining bill is written down and accepted.

Considerations to state explicitly, not defaults; each one is a claim the next experiment should test.

What to expect before you run2 minPublished priors for tabular data and images, and why they do not replace your own comparison.

Priors for choosing what to compare Overview

Tabular business data
Tree ensembles are hard for small networks to beat in published comparisons.
Images
Convolutional networks use the pixel layout; a flat MLP ignores it (Day 36).
A prior
The executed comparison on your split decides.
Running cost
A pipeline and a server must be paid for by a gain larger than the spread.

Grinsztajn et al. (2022) and Shwartz-Ziv and Armon (2022) for the tabular finding; both use their own benchmark suites.

Memory anchor

Same rows, same metric, same effort, several seeds: then write the claim at the strength the evidence supports.

Useful for

Writing the results paragraph of a neural-versus-classical experiment so that every sentence is backed by a printed number.
Reviewing a colleague's "the network wins" claim - on fraud scores, claims triage or churn alike - and listing what would have to be true for it to hold.
Deciding what to run next (more seeds, a tuned baseline, the sealed test part) before spending more compute.

Not suitable for

Establishing production performance - that needs the sealed test evaluation, then monitoring for drift and skew (Days 18, 19 and 29).
Choosing an architecture for images, text or large data - the protocol transfers, but a 64-32-10 numpy network is a teaching object, not a candidate.

Traps

A tuned network against a default baseline

The network gets a search over learning rate, width and epochs; the baseline runs at its defaults. The network "wins" by an amount that tuning the baseline would have removed or reversed.

How to avoid it

Give both models the same tuning budget on training rows, report the settings and the number of trials for each.

One split, one seed, one number

A gap of a fraction of a point is reported as a ranking. On another split or initialization the order changes, and the team has built a pipeline on noise.

How to avoid it

Report the standard error, the paired disagreements and the spread over at least five splits or seeds before ranking.

Senior reasoning

A senior writes the claim last. They list what was held constant, what was not (tuning effort, seeds, sample size), and state the claim at the strength those facts allow: "tied on accuracy, lower log loss on one split with an untuned baseline" is a different sentence from "the network is better". They treat the sealed test part as a one-use instrument, prefer the simpler model when the evidence is a tie, and say which single further experiment would change their mind.

Interview question

"Your network gets 96.4% and the logistic regression 96.7%. What do you tell the manager?" How would you answer?

Outline your answer first.

Recall cards scheduled after completion

What must be identical, and what must be comparable, in a fair network-versus-baseline comparison?
Two models score 0.964 and 0.967 accuracy on 360 rows. What do you say?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Accounting for Variance in Machine Learning Benchmarks — Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Naz Sepah, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Dmitriy Serdyuk, Tal Arbel, Chris Pal, Gael Varoquaux, Pascal Vincent, 2021core
Reports that data sampling, parameter initialization and hyperparameter choice change the conclusions of benchmark comparisons, supporting repeated splits and seeds.
Why do tree-based models still outperform deep learning on tabular data? — Leo Grinsztajn, Edouard Oyallon, Gael Varoquaux, 2022optional
On 45 tabular datasets, tree-based models remained strong against neural networks on medium-sized data, supporting a gradient-boosting baseline before a network.
Tabular data: Deep learning is not all you need — Ravid Shwartz-Ziv, Amitai Armon, 2022optional
Finds XGBoost outperformed several deep tabular models on the datasets studied and needed far less tuning.
Common pitfalls and recommended practices (scikit-learn User Guide, section 12) — scikit-learn developerscore
Official guidance on leakage from preprocessing and on keeping transformers inside the training fold.
Inference for the Generalization Error — Claude Nadeau, Yoshua Bengio, 2003optional
The corrected resampled t-test, which accounts for overlapping training sets when results are repeated over splits.
scikit-learn load_digits: optical recognition of handwritten digits (bundled copy) — scikit-learn developersoptional
Identity and size of the bundled digits dataset (1,797 samples, 64 features, 10 classes) used in the block.
A Recipe for Training Neural Networks — Andrej Karpathy, 2019optional
Practitioner checklist for verifying a network (initial loss, overfit a batch) before trusting any comparison; cited as experience, not as a study.

Optional sections

Deep dive · 10 min

Paired comparison. Two accuracies computed on the same rows are positively correlated, so their difference has a smaller standard error than the independent formula suggests. What matters is the discordant pairs: rows only one model gets right. Here the network is right alone on 1 row and logistic regression on 2. Under the null of equal accuracy, those 3 discordant rows split like fair coin flips, and an exact two-sided binomial test gives a p-value of 1.0. McNemar's test is this idea. The same pairing works for log loss: take the per-row loss difference and look at its mean and spread.

Why log loss moved and accuracy did not. Accuracy only counts argmax decisions. Log loss rewards well-scaled confidence, so a model can improve its probabilities (for instance through regularization strength) without changing many decisions. That is why the two metrics can disagree, and why the choice between them should follow the decision: counting correct readings favours accuracy, thresholding or cost weighting favours log loss and calibration.

What a sealed test part is for. It gives one unbiased estimate of the single configuration you chose, so it can be used once. Looking at it repeatedly to pick among candidates makes it another validation set, and the final number is then optimistic by an unknown amount.

Scope of the lab. Five splits of 1,437 development rows each is a small experiment. It can show that a gap is not stable; it cannot tell you the true size of the gap, and it says nothing about different data, a CNN, or a larger tuning budget for the network.

Hands-on lab: Five splits, a tuned baseline and a gradient-boosting reference · 20 min

Run the block (under a minute on a laptop CPU). On each of five stratified splits it trains the same numpy network, selects the logistic regression's regularization strength by 5-fold cross-validation on training rows only, and fits a histogram gradient-boosting model. It prints mean and standard deviation of validation accuracy and log loss for the three models, and the per-split log-loss difference between the network and the tuned logistic regression.

Predict before running: on how many of the five splits will the network have the lower log loss? Then answer: (1) How does the sign of the log-loss gap compare with the core block's single split, and what changed in the baseline? (2) Are the accuracy differences between the three models larger than their split-to-split standard deviations? (3) Write a two-sentence result statement that quotes only these printed numbers. (4) Give the network the same search effort (a small grid over width and learning rate chosen by cross-validation on training rows) and re-run; what would you now be able to claim?

import os
os.environ["OMP_NUM_THREADS"] = "1"  # keeps HistGradientBoosting from oversubscribing a shared machine
import numpy as np
from sklearn.datasets import load_digits
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss
from sklearn.model_selection import GridSearchCV, train_test_split

X, y = load_digits(return_X_y=True)
X = X / 16.0

def softmax(Z):
    E = np.exp(Z - Z.max(1, keepdims=True))
    return E / E.sum(1, keepdims=True)

def mlp_val_probs(Xtr, ytr, Xva, seed, hidden=32, lr=0.5, epochs=300):
    r = np.random.default_rng(seed)
    W1, b1 = r.normal(0, np.sqrt(2 / 64), (64, hidden)), np.zeros(hidden)
    W2, b2 = r.normal(0, np.sqrt(1 / hidden), (hidden, 10)), np.zeros(10)
    T = np.eye(10)[ytr]
    for _ in range(epochs):
        H = np.maximum(0, Xtr @ W1 + b1)
        dZ = (softmax(H @ W2 + b2) - T) / len(Xtr)
        dH = (dZ @ W2.T) * (H > 0)
        W2, b2 = W2 - lr * H.T @ dZ, b2 - lr * dZ.sum(0)
        W1, b1 = W1 - lr * Xtr.T @ dH, b1 - lr * dH.sum(0)
    return softmax(np.maximum(0, Xva @ W1 + b1) @ W2 + b2)

rows = []
for seed in range(5):                                   # five different splits, same recipe
    dev_X, _, dev_y, _ = train_test_split(X, y, test_size=0.2, stratify=y, random_state=seed)
    Xtr, Xva, ytr, yva = train_test_split(dev_X, dev_y, test_size=0.25, stratify=dev_y, random_state=seed)
    grid = GridSearchCV(LogisticRegression(max_iter=2000), {"C": [0.1, 1, 10, 100]}, cv=5, scoring="neg_log_loss")
    models = {
        "mlp": mlp_val_probs(Xtr, ytr, Xva, seed),
        "logreg_cv": grid.fit(Xtr, ytr).predict_proba(Xva),
        "hist_gb": HistGradientBoostingClassifier(random_state=seed).fit(Xtr, ytr).predict_proba(Xva),
    }
    rows.append({k: (np.mean(p.argmax(1) == yva), log_loss(yva, p, labels=range(10))) for k, p in models.items()})

for k in ("mlp", "logreg_cv", "hist_gb"):
    acc = np.array([r[k][0] for r in rows]); ll = np.array([r[k][1] for r in rows])
    print(f"{k:<10} accuracy {acc.mean():.3f} +/- {acc.std(ddof=1):.3f} | log loss {ll.mean():.3f} +/- {ll.std(ddof=1):.3f}")
d = np.array([r["mlp"][1] - r["logreg_cv"][1] for r in rows])
print("MLP minus tuned logistic regression, log loss per split:", np.round(d, 3))
print(f"MLP better on {np.sum(d < 0)} of 5 splits; mean difference {d.mean():.3f}, sd {d.std(ddof=1):.3f}")

Interview practice · 10 min

A teammate presents a network that beats gradient boosting by 0.8 points of AUC on one validation split of a tabular churn dataset and asks to deploy it. What is your review?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-18 · depends on numpy, scikit-learn