Core day 56 of 60 · Week 12 · Part II · evaluation depth

Neural Network Project

Your Weeks 7-8 neural project reports a small network and a classical baseline on one frozen split. A reviewer asks: what does this comparison prove, and what must you show before they accept that the network is better? Using your own run log, what is the strongest sentence your evidence supports?

By the end: From your own neural-project artefacts you write the claim you wanted to make, map each rung of the evidence ladder to a file or run row, mark the rungs that are not shown, and rewrite the claim as the strongest sentence the shown rungs support.

At a glance

Map every rung of the evidence ladder to an artefact, report seed noise and evaluation noise separately, and cut the claim to what the shown rungs support.

Five rungs
Correct, fair, larger than noise, relevant, bounded. The lowest unsupported rung caps the claim.
No artefact, no credit
A file path or a run-log row per rung; memory does not count.
Two noise sources
Seeds = retraining noise. Block bootstrap = which weeks were sampled. Neither contains the other.
Test opened once
A test number used to choose anything is a validation-level number.
A tie is a finding
Recommend the simpler model and name the evidence that would reopen it.

Recall · 5 min

Without opening your files, answer in one line each: (1) Which four things must be identical for a neural-versus-baseline comparison to be fair? (2) What should the loss at initialisation be for a balanced binary task with a zero output bias, and what did your run print? (3) Which data is allowed to choose your final configuration? Then clear your due review cards.

Open due reviews

Concept · 10 min

Today is a review day, not a new technique. Your Weeks 7-8 artefacts hold the results: frozen split, baseline, run log, ablation table, report. Read them as a stranger would. A comparison is a claim, and a claim is only as high as its lowest rung with an artefact behind it. A well-evidenced tie beats a win with two rungs.

The gap between a tiny network and its baseline is only meaningful beside the seed-to-seed spread

Predict first

Changing only the seed moves the network's validation loss by 0.006, and its gap to the baseline is 0.002. What does that say?

Text description of the figure

A tiny multilayer perceptron trained next to a baseline on one split, with controls for hidden units, learning rate, epochs and seed. Changing only the seed moves the validation loss, which you compare with the size of the gap between the two models.

The claim ladder: read it from the bottom up Stack

5 BoundedYou can say where the result stops holding: slices, periods, data size.
4 RelevantThe gap matters for a decision, in its units, and holds on the slices that matter.
3 Larger than noiseGap beyond seed-to-seed spread and beyond evaluation-sample uncertainty.
2 FairSame split, features, metric and tuning budget; the test part opened once.
1 CorrectShape assertions, gradient check, tiny-batch overfit, loss at initialisation.

Each rung needs a named artefact. The claim may not rise above the first rung that is "not shown".

The review in four moves Sequence

Write the claim
One sentence with numbers: the claim you would like to make.
One artefact per rung
A file path, a run-log row or a plot. Missing means not shown.
Separate the noise
Seed spread for retraining; a block bootstrap for the evaluation sample.
Cut the claim
The strongest sentence the shown rungs support, plus what would change it.

Two noise sources, computed on real training runs Illustration

Mean 0.397, seed sd 0.014; 1 of 10 seeds beat the baseline. Report only the best seed and the network "wins".

Each line is honest about which blocks were sampled, for one network. Their centres span 0.047, against a mean interval width of 0.042: retraining moves the answer, and no single interval shows it.

Computed: a 2-6-1 tanh MLP (Adam, 50 epochs) and logistic regression on the same seeded two-moons split (100 validation rows cut into 10 blocks as stand-ins for weeks; 1,000 block resamples per seed). Synthetic data, not a project result.

Left, retraining noise across seeds. Right, one evaluation interval per trained network. Neither picture contains the other.

Two noise sources, two different questions Comparison

Criterion	Seed spread	Block bootstrap
Question it answers	How much does the metric move if I train again?	How much does it move if the evaluation weeks were drawn again?
How you measure it	Several seeds (initialisation, batch order); sample sd	Resample weekly blocks of one trained model's errors
Covers retraining	Yes	No
Covers the sampled weeks	No	Yes

Report both lines. More bootstrap resamples tighten the same quantity; they never add the seed variation.

Application output: the gaps against the noise (log loss, toy numbers) Chart

Validation gap0.42 seed sd0.0022
One seed sd0.0053
Rule: 2 seed sdthe bar written before the runs0.0106
Test gapinterval 0.0029 to 0.0137 over weeks; 1.62 seed sd0.0085
Shifted-route slicenetwork worse here-0.006

Positive means the network has the lower loss. Synthetic stand-ins for your run log, printed by the application code.

Three reviewer checks on the toy run log Table

Check	Result	What it means
Validation gap above 2 seed sd	Fail (0.42 sd)	Training noise alone could produce the gap
Test interval over weeks excludes zero	Pass (0.0029 to 0.0137)	For this one trained network, on these weeks
Not worse on the shifted slice	Fail (-0.006)	The gain reverses where conditions change
Recommend the network?	No	Recommend the cheaper model; list what would reopen it

Key terms: Claim ladder, Baseline, Seed spread, Block bootstrap, Paired comparison, Test-set reuse.
Four moves: write the claim you want; find an artefact for every rung (no artefact, no credit); separate the two noise sources; cut the claim to the strongest sentence the shown rungs support.
Seeds measure training noise; a block bootstrap over weeks measures evaluation-sample noise for the one network you trained. Report both.
A baseline comparison proves only what it measured: this data, split, features, tuning budget and metric. Trees remain a strong prior on medium-sized tabular data (Grinsztajn et al., 2022).
Evidence lives on /portfolio: one Experiments row per final run, the choice in the Decision log, the write-up under Project reports. A reference implementation is never your evidence.

Assumptions

Your thread has a frozen split, a baseline and run records from Weeks 7-8; anything missing is reported as "not shown", not reconstructed.
Seeds and blocks are the only noise you can measure; a different data draw or a different tuning budget stays unmeasured unless you ran it.
The test part was opened once per model; if it was opened more than once, the test number is a validation-level number.

Application and workplace decision · 10 min

(a) Write the claim you would make today, for example "the network beats the baseline by X on test log loss".
(b) Build a five-row table, one row per rung: artefact (path or run-log row), status (shown, partly, not shown), the one thing missing.
(c) Rewrite the claim as the strongest sentence the table supports; save it in the Decision log with alternatives, evidence and a "reconsider when" line.
Then run the code. Its numbers are toy stand-ins for your run log; it applies three reviewer checks (validation gap against seed spread, a block-bootstrap interval on the test year, a shifted slice). Replace the arrays with your own CSV.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np

# Toy stand-ins for YOUR run log (synthetic numbers, not a real result): validation log loss per seed.
mlp = np.array([0.536, 0.541, 0.532, 0.546, 0.539])  # 5 initialisation / batch-order seeds
gbm = np.array([0.541, 0.540, 0.543, 0.539, 0.542])  # boosting with row subsampling, 5 seeds
gap = gbm.mean() - mlp.mean()  # positive = the MLP has the lower loss
sd = max(mlp.std(ddof=1), gbm.std(ddof=1))  # training noise: seed-to-seed spread
print(f"validation: gap {gap:.4f}, seed sd {sd:.4f}, gap/sd {gap / sd:.2f}")

# One test year as 52 weekly blocks: paired per-week difference (gbm - mlp) in mean log loss.
rng = np.random.default_rng(56)
week_diff = rng.normal(0.003, 0.02, 52)
boot = np.array([rng.choice(week_diff, 52).mean() for _ in range(2000)])
lo, hi = np.percentile(boot, [2.5, 97.5])  # evaluation-sample noise, one trained network
print(f"test: gap {week_diff.mean():.4f}, 95% block-bootstrap [{lo:.4f}, {hi:.4f}], gap/sd {week_diff.mean() / sd:.2f}")

shifted_gap = -0.006  # toy: on the shifted-route slice the MLP is worse than GBM
checks = {
    "validation gap > 2 seed sd": gap > 2 * sd,
    "test interval excludes zero": lo > 0 or hi < 0,
    "not worse on shifted slice": shifted_gap >= 0,
}
for name, ok in checks.items():
    print(f"  {'pass' if ok else 'FAIL'}  {name}")
print("recommend MLP:", all(checks.values()))

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 32 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

What a comparison estimates: one gap, two variances4 minThe estimand behind "model A beats model B", and the variance decomposition that says which replication shrinks which term.

One observed gap, two random draws: each replication shrinks one term Hierarchy

Observed gap Δ̂one training run, one test sample
- retrainTraining drawinitialisation, batch order
  - more seeds Sσ²_seed / S shrinksestimate: variance of the per-seed metric
- redraw the weeksEvaluation drawwhich weeks were sampled
  - more blocks Bσ²_block/B shrinksestimate: variance of per-block gaps
  - more resamplesNo term shrinksonly the block term is estimated better

The estimand is Δ = E[L_base − L_net] over both draws; a report shows one realization of each. Only a new training run or new evaluation data moves a term; bootstrap resamples reuse the same data.

Why more seeds do not shrink everything (two-component model) Formula

\operatorname{Var}\left(\hat\Delta\right) \approx \frac{\sigma^{2}_{\text{seed}}}{S} + \frac{\sigma^{2}_{\text{block}}}{B}

$\hat\Delta$: gap between the models, averaged over S seeds and scored on B shared evaluation blocks
$\sigma^{2}_{\text{seed}}$: retraining variance; only more seeds shrink this term
$\sigma^{2}_{\text{block}}$: variance of the paired per-block gap; only more evaluation blocks shrink it
$S, B$: five seeds cannot rescue a test year of twelve blocks, and vice versa

A simplifying model that assumes the two sources are independent. It explains why reporting only one of them is never the whole uncertainty.

With the toy numbers the two terms are the same size: neither can be skipped Bridge

Seed term 0.0053² / 55.6 × 10⁻⁶
Block term 0.02² / 5252 weekly blocks of one trained network+7.7 × 10⁻⁶
Variance of the gap13.3 × 10⁻⁶

Variance contributed to the gap by each source, from the application's synthetic run log. The seed term rests on only five seeds, so it is also the worse-known one.

The decision rule from project 028, written before the runs Formula

\frac{\bar L_{\text{baseline}} - \bar L_{\text{network}}}{s_{\text{seed}}} > 2

$\bar L$: mean validation log loss over five seeds
$s_{\text{seed}}$: sample standard deviation (divisor n - 1) across seeds; write down whose
$> 2$: the bar was fixed in advance; a threshold chosen after the numbers is a description

The rule also requires that the network is not worse on the shifted-route slice.

Paired tests: subtract before you test5 minThe paired t on per-block differences, worked on six weeks by hand, and what the same data say when the pairing is thrown away.

The paired t on per-block differences Formula

d_b = L_b^{\text{base}} - L_b^{\text{net}}, \qquad t = \frac{\bar d}{s_d / \sqrt{B}}, \qquad \bar d \pm t_{B-1,\,0.975}\,\frac{s_d}{\sqrt{B}}

$d_b$: one number per evaluation block b (a week): both models scored on the same rows, then subtracted
$\bar d,\; s_d$: mean and sample standard deviation (divisor B - 1) of the B differences
$s_d/\sqrt{B}$: the standard error of the mean difference; only more blocks shrink it
$t_{B-1}$: Student t with B - 1 degrees of freedom: 2.571 for B = 6, 2.01 for a 52-week year

Subtracting first removes everything the two models share - week difficulty, seasonality, mix - leaving only the disagreement. Assumes roughly independent, not-too-heavy-tailed differences; blocks exist to make that credible.

Worked by hand: six weekly blocks of a telecom churn challenger (illustrative) Table

Week	GBM loss	MLP loss	d = GBM - MLP	d - mean	(d - mean)^2 x 1e6
1	0.521	0.510	+0.011	+0.005	25
2	0.498	0.502	-0.004	-0.010	100
3	0.544	0.535	+0.009	+0.003	9
4	0.512	0.499	+0.013	+0.007	49
5	0.533	0.531	+0.002	-0.004	16
6	0.506	0.501	+0.005	-0.001	1

Mean d = 0.0060; variance 200e-6 / 5, so s_d = 0.00632 and SE = 0.00258. t = 2.32 against t_5 = 2.571: the 95% interval [-0.0006, +0.0126] touches zero (p = 0.068). Six good weeks are not yet evidence. Verified with scipy ttest_rel.

Why pairing wins: the models stumble on the same weeks Illustration

Weekly loss swings by 0.046 from the easiest to the hardest week, and the two models move together (correlation 0.93). An unpaired test counts that swing as noise.

Subtract first: the six differences have mean 0.0060 and sd 0.0063, against a weekly-loss sd of about 0.017. The dashed gaps on the left are these bars.

Illustrative numbers: the six weeks of the worked table above (a hypothetical telecom churn challenger). Both panels are drawn from the same twelve losses.

Var(d) = Var(L_GBM) + Var(L_MLP) − 2·Cov(L_GBM, L_MLP): shared week difficulty makes the covariance large, and subtracting cancels it. The same algebra holds for two triage rules on the same hospital weeks, two fraud models on the same card days, two rankers on the same cohorts.

Same six weeks, with and without the pairing Table

Analysis	Standard error	t	p-value	What it treats as noise
Unpaired (Welch) on the 6 + 6 weekly losses	0.0096	0.63	0.55	Week-to-week difficulty counts as noise
Paired t on the 6 differences	0.00258	2.32	0.068	Shared week difficulty subtracted out first

Weekly losses swing by about 0.017 (sd) while the models never disagree by more than 0.013, so the unpaired SE is 3.7 times larger on the same data and the signal drowns. Always difference first. Computed with scipy on the table above.

The block bootstrap, step by step5 minThe algorithm, three resamples done by hand, where the percentile interval comes from, and when not to trust it.

The block bootstrap, precisely Sequence

Cut into blocks
Split the evaluation period into B contiguous blocks (weeks) long enough that dependence stays inside a block.
One number per block
The paired mean gap d_b per block, from one trained model per side.
Resample B blocks with replacement
Some weeks appear twice, some not at all; whole blocks move together, which preserves within-week correlation.
Average
The mean of the resampled d's is one draw; 1,000-2,000 repeats trace the sampling distribution.
Read the percentiles
The 2.5th and 97.5th percentiles of the recorded means form the 95% interval.

Three resamples of the six worked differences, by hand (seed 56) Table

Resample	Blocks drawn (values of d)	Mean
1	+0.009, +0.002, +0.009, +0.011, +0.005, +0.005	+0.0068
2	+0.013, +0.002, +0.011, +0.013, -0.004, -0.004	+0.0052
3	+0.009, +0.013, +0.011, +0.009, +0.005, +0.013	+0.0100

Six draws with replacement, averaged, 2,000 times: interval [+0.0010, +0.0103], narrower than the t-interval and excluding zero. With B = 6 the bootstrap undercovers; under 15-20 blocks trust the t. numpy, default_rng(56).

The algorithm, watched: resamples and the percentile interval Illustration

Each resample keeps whole weeks, so within-week correlation survives; stacked dots are weeks drawn twice or more. The three resample means (+0.006, -0.001, +0.002) already hint at the spread.

The 95% percentile interval is [-0.005, +0.009] around the observed +0.002. It covers which weeks were sampled, for this one trained model; it says nothing about retraining.

Computed: 12 weekly paired gaps drawn once from a seeded normal (mean 0.004, sd 0.012) as stand-ins for a 52-week year, then 2,000 block resamples of those same 12 values. Synthetic data, not a project result.

Left, three literal resamples of twelve weekly blocks (stacked dots are weeks drawn twice). Right, 2,000 resample means and the percentiles that become the interval.

Four ways a block-bootstrap interval misleads, and the fix for each Hierarchy

Block bootstrapwhole weeks move together
- rows, not weeksToo narrowone week's rows share weather, campaigns
- under 15-20 blocksUndercoversuse the paired t-interval instead
- dependence spans weeksToo narrow againmake the blocks longer
- the process driftsSplit the periodno resampling fixes drift

Rows of one week share campaigns, weather and outages, so a row bootstrap pretends you have far more independent data than you do. The six-week worked table is the undercovering case.

One interval that carries both noise sources (two-level bootstrap) Sequence

Train S seeds per model
Same split, features and budget; keep every seed's per-row predictions on the evaluation part.
Paired per-block gaps
For each seed, the mean loss difference per evaluation block (week).
Resample seeds
Draw S seeds with replacement; this carries retraining noise.
Resample blocks
Draw B blocks with replacement; the same blocks for both models keep the pairing.
Read the percentiles
Mean gap per draw; 2.5th and 97.5th percentiles. With S = 5 the seed level stays coarse: say so.

The only single interval that carries both variance terms. With five seeds its seed level is coarse, so the report says so next to the interval.

Rung 1: is the network doing what you think?4 minFour sanity checks, the loss to expect before training, and the eight-row test that catches what the loss check misses.

Four checks before any comparison Overview

Shape assertions
Every weight, bias and activation has the shape you wrote down (Day 31).
Gradient check
Analytic against numerical gradient; small relative error (Day 33).
Tiny-batch overfit
A handful of rows can be fitted to near-zero loss; if not, the model cannot learn.
Initial loss
Matches ln C before training starts.

The Correct rung: loss at initialisation Formula

L_0 = -\ln\left(\tfrac{1}{C}\right) = \ln C

$C$: number of balanced classes; outputs near uniform at initialisation
$C = 2$: ln 2 = 0.693 for a balanced binary task with a zero output bias
$L_0$: expected first-batch cross-entropy; far above it points to a bug in labels, scaling or the output layer

Rung 1 in practice: initial loss and the eight-row overfit test Illustration

ln 2 holds when the outputs start near 0.5. Means: small output weights 0.70; default (Glorot) init 0.80; bug: output bias +3 1.62. A start far above ln 2 is a bug to find before tuning.

A network that cannot memorise eight rows cannot learn the task. Both bugs pass the initial-loss check; only this one catches them.

Computed with the course MLP code (2-6-1, tanh, Adam) on a seeded two-moons training set (100 rows; the first 8 for the overfit test). The bugs are deliberate.

Two checks that take seconds. The second catches bugs the first one cannot see.

Rung 2: a fair contest4 minWhat must be identical, how to tell a broken estimate from an unfair one, and how much selection on test costs.

The Fair rung: five things that must be identical Comparison

Criterion	Fair	Unfair example
Split	Same frozen rows and dates for both models	Network on a later split than the baseline
Features	Same inputs; preprocessing fitted on training rows	Extra engineered features for one model
Metric	Same metric on the same scale	Log loss for one, accuracy for the other
Tuning budget	Comparable search for both	A wide network search against boosting defaults
Test part	Opened once per model, after all choices	Re-scored after every tuning change

Every row must match before a gap between the two models means anything; write the matching artefact next to each row.

What kind of number is your test result? Diagram

Test used to choose?
- yes: to Validation-level number
no
Equal tuning budgets?
- no: to Unfair comparison
yes
Test-grade numbernow check noise
Validation-level numberoptimistic: say so
Unfair comparisonan unbiased number, a skewed contest

Selection on the test part breaks the estimate itself; an unequal budget only limits what the estimate may claim.

Picking the best test score makes it optimistic Illustration

Every configuration is equally good (true loss 0.400). Keeping the best of 10 test scores reports 0.382 on average: 0.018 too optimistic, from selection alone.

The whole distribution shifts, not just the average. A fresh, untouched test part would centre on 0.400 again.

Seeded simulation: 2,000 repetitions; each test score is the true loss plus normal noise with sd 0.012. Illustrates selection bias, not a project result.

Identical configurations, one noisy test part. Selection alone moves the reported number.

Hidden randomness in the librariesadvanced3 minWhich defaults add noise you did not plan for, verified against the installed versions.

Hidden randomness and defaults (scikit-learn 1.9.1, NumPy 2.5.3, pandas 3.0.6) Table

Setting	Default	Why the review cares
MLPClassifier random_state	None	Init and batch order change each run: set and vary seeds
MLPClassifier batch_size	auto = min(200, n_samples)	Batch order is a second noise source next to initialisation
MLPClassifier early_stopping	False (validation_fraction 0.1 when on)	When on, its internal split is another random choice
HistGradientBoosting early_stopping	auto: on when n_samples > 10,000	Above that size boosting is not seed-free either
np.std(x)	ddof = 0 (population)	Use ddof=1 for a seed spread; five seeds make the difference visible
pandas Series.std()	ddof = 1 (sample)	Two libraries, two defaults: write which one the rule used

Checked with get_params() and the library source in this repository's Python environment. Re-check after an upgrade.

When an expert would not reach for the network3 minWhat decides between boosted trees and an MLP on tabular data, beyond one score.

Network or boosted trees on tabular data: what decides it Comparison

Criterion	Boosted trees	MLP
Prior on medium tabular data	Strong (Grinsztajn et al., 2022)	Must earn it on your split
Preprocessing	No scaling; ordinal or native categorical handling	Scaling and imputation fitted on training rows; embeddings for categories
Missing values	Handled natively (HistGradientBoosting, XGBoost, LightGBM)	Needs imputation plus a missing indicator
Monotone constraints	Supported (monotonic_cst, monotone_constraints)	Not built in; needs a constrained architecture
Where it tends to earn its cost	Heterogeneous features, irregular targets, small to medium data	Text, images or sequences next to the table; very large data; shared representations

The last row is practitioner judgement, not a benchmark result. Your own comparison is the evidence for your data.

Rungs 4 and 5, the staff-level review and where evidence livesadvanced4 minDecision units, slices and triggers; the sentences a senior reviewer replaces; the /portfolio places a reviewer checks.

Rungs 4 and 5: relevant and bounded Sequence

Decision units
The gap in what the business pays for (orders sent to manual review, claims to an adjuster, cases flagged per week) against serving cost.
Slices
Where conditions change - newly onboarded merchants, a new product category, peak weeks - does the gain hold or reverse?
Limits
The data size, period and features the result was measured on; what was never tested.
Triggers
The dated evidence that would reopen the recommendation: more history, a stable shifted slice, a change in cost per case.

Five common readings, and what a staff-level reviewer says instead Comparison

Criterion	Common reading	Staff-level reading
Interval excludes zero	So the network is better	For this trained network on these blocks; retrain and the centre moves
Five seeds	So the spread is known	A rough spread: report it with its n; do not over-read the second decimal
Equal epochs	So the budget was fair	Budget is configurations tried and compute spent, for both models
A tie	So the project failed	With every rung shown, recommend the cheaper model and name what reopens it
Test used once at the end	So the estimate is clean	Ask how many times it was opened and what changed after each look

Where a reviewer looks on /portfolio Overview

Experiments form
One row per final run: model, seed, split, metric, value.
Decision log
The recommendation, alternatives, evidence and a reconsider-when line.
Project reports
The write-up with the supported sentence and the not-shown rungs.
Portfolio review
The proof your neural thread still lacks, listed at the top of the page.

A reference implementation is never credited as your evidence. Reference r06 is marked implemented with recorded results: read them only after you have written your own verdict.

Memory anchor

A comparison is a claim on a ladder of five rungs - cut the claim to the lowest rung with an artefact.

Useful for

Reviewing any model-versus-baseline comparison, yours or a colleague's, before it enters a decision.
Deciding what proof your neural-project thread still lacks before you present it.
Writing a conclusion that holds when the result is a tie.

Not suitable for

Proving that a model family is better in general; that needs several datasets and a pre-registered protocol.
Replacing the training and debugging work of Weeks 7-8; a review can only report which checks were never run.

Traps

Reconstructing evidence from memory

A missing gradient check or seed table is "remembered" during review, so the rung is marked shown without an artefact. A reviewer who asks for the file ends the conversation.

How to avoid it

Mark the rung not shown, state it plainly in the report and add the check to the plan.

Reporting an interval for the wrong noise

A block-bootstrap interval around one trained network is presented as if it covered retraining, so a small gap looks decisive.

How to avoid it

Report seed spread and evaluation-block interval as two separate lines and say what each covers.

Senior reasoning

A senior reviews their own project as an adversary would. They write the claim first, mark which rungs have artefacts, then cut the claim to match. They report ties as findings, keep the cost of the more complex model in view, and leave a decision record that names the evidence which would change the recommendation, so the project stays useful after the author has moved on.

Interview question

"Walk me through your neural-network project. What did the comparison with the baseline prove, and what did it not?" How would you structure the answer?

Outline your answer first.

Recall cards scheduled after completion

Name the five rungs of the evidence ladder for a neural-versus-baseline claim.
What do seeds measure, and what does a block bootstrap over weeks measure?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Why do tree-based models still outperform deep learning on tabular data? — Leo Grinsztajn, Edouard Oyallon, Gael Varoquaux, 2022core
Evidence that tree-based models remain strong on medium-sized tabular data, which sets the prior your own comparison tests. (Abstract and results)
Accounting for Variance in Machine Learning Benchmarks — Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Naz Sepah, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Dmitriy Serdyuk, Tal Arbel, Chris Pal, Gael Varoquaux, Pascal Vincent, 2021core
Data sampling, initialisation and hyperparameter choice each change benchmark conclusions, which is why seed spread and evaluation noise are reported separately.
CS231n course notes: Neural Networks Part 3 - Learning and Evaluation (gradient checks, sanity checks, babysitting learning) — Stanford CS231n course staffoptional
The correctness rung: gradient check with relative error, loss at chance and overfitting a tiny subset. (Gradient checks; sanity checks before learning)
On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation — Gavin C. Cawley, Nicola L. C. Talbot, 2010optional
Selecting on the evaluation data produces an optimistic performance estimate, the finding behind question q-056-3.

Optional sections

Deep dive · 10 min

A decision rule written before the runs. Project 028 proposes recommending the network only if its mean validation log loss over five seeds is lower by more than two seed standard deviations and it is not worse on the shifted-route slice. The rule is useful because it was fixed before the comparison; a threshold chosen after seeing the numbers is a description, not a rule. Which standard deviation to use (the network's, the larger of the two) is a choice; write it down.

Two noise sources, one table. With $S$ seeds and $B$ evaluation blocks, the seed spread estimates how the metric varies when training is repeated, and the block bootstrap estimates how it varies when the evaluation data are drawn again. A paired design (same weeks for both models) removes week-to-week difficulty from the difference. If the network is retrained per seed, average the per-seed predictions or bootstrap the seed-averaged loss to include both sources.

The reference implementation. Where a reference implementation exists, for example the one listed for project 028, check its registry status first. Reference r06 is marked "implemented" and has recorded results (the MLP lost to logistic regression and boosting on that synthetic data); read them only after you have written your own verdict, and note that the scoring policy never credits reference work as your evidence.

Interview practice · 10 min

Your network beats gradient boosting by 0.004 log loss on the test year, but the gap is smaller than the seed-to-seed spread. A manager wants "the AI model" in production. What do you say?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-18 · depends on numpy

1 Recall · 5 min

2 Concept · 10 min

Assumptions

3 Application and workplace decision · 10 min

4 Mini-assessment · 8 min

Technical depth · optional · 32 min in total

Useful for

Not suitable for

Traps

Senior reasoning

Interview question

Recall cards scheduled after completion

Prerequisites and related concepts

References

Optional sections

Recall · 5 min

Concept · 10 min

Application and workplace decision · 10 min

Mini-assessment · 8 min