project-034 · Deep learning · cross-industry

Is it the learning rate or the seed? A repeated-seed ablation of learning rate, weight decay and dropout

A team standardising a small-network template keeps getting conflicting results from single training runs. Which learning-rate and weight-decay settings train a small MLP reliably on a small image-like dataset, and how large must a difference between two settings be before it is more than seed noise?

Dataset: scikit-learn digits (8×8 handwritten digits)
Licence: CC BY 4.0 per the UCI source page (dataset-uci-optical-digits); redistributed inside scikit-learn
Reference run: executed
Lessons: day 33 day 34 day 35 day 56

Data
scikit-learn digits (8×8 handwritten digits) (bundled-with-library) · 1,797 images × 64 features, 10 classes of about 180 each
Details on data
Baseline
Majority class (about 10% accuracy) and multinomial logistic regression on the same split.
Details on baseline
Validation
One stratified 60/20/20 train/validation/test split with a fixed split seed. In the grid, seeds change only initialisation and mini-batch order, so the spread you see is…
Details on validation
Metric
Validation log loss and accuracy (mean ± sd, n = 5 per cell), divergence count (NaN or loss above 10), generalisation gap (train minus validation accuracy), epochs to…
Details on metric

Success criterion

A 5 × 4 grid (learning rate × weight decay) with 5 seeds per cell (at least 3 required): mean ± sd of validation log loss and accuracy, and the number of diverged runs per cell.
A selection rule written down before running: lowest mean validation log loss; cells within one pooled sd count as ties, broken toward the setting further from the divergence region.
The chosen setting's test result, read once.
A written conclusion that explains the learning-rate × weight-decay interaction, the size of seed noise, and how many seeds a future comparison needs.
Homework: a dropout ablation (3 rates × at least 3 seeds) with the same reporting.

Context

Days 33-35 and the Day 56 neural project. Single runs mislead because initialisation, data order and data sampling all move results (Bouthillier et al., 2021). The reference implementation r07 runs in numpy on the bundled digits data in a few minutes on CPU. It uses plain mini-batch SGD on purpose: with plain SGD an L2 penalty in the loss and weight decay in the update are the same thing up to a rescaling by the learning rate, while with adaptive optimisers such as Adam they differ (Loshchilov and Hutter, 2019). Stating which one you ran is part of the result.

Data

Dataset: scikit-learn digits (8×8 handwritten digits) (bundled-with-library)
Licence: CC BY 4.0 per the UCI source page (dataset-uci-optical-digits); redistributed inside scikit-learn
Size: 1,797 images × 64 features, 10 classes of about 180 each
Verification: executed

Access

sklearn.datasets.load_digits(); ships inside scikit-learn, no network and no authentication.

Grain

One row = one handwritten digit, downsampled to an 8×8 grid of block counts (0-16).

Column	Meaning
pixel_0_0 ... pixel_7_7	Count of 'on' pixels in each 4×4 block of the original 32×32 bitmap, integer 0-16.
target	Digit label 0-9.
images	The same data reshaped to (1797, 8, 8) for visual checks.

EDA and data-quality rules

Confirm shape (1797, 64), integer range 0-16 and class counts (about 180 each) and assert them in code.
Hash rows to find exact duplicates and report them; keep them on one side of the split.
Writer identity is not available, and this copy comes from 13 writers in the UCI test set, so a random split mixes writers; note that results describe these writers, not new handwriting.

Baseline

Majority class (about 10% accuracy) and multinomial logistic regression on the same split. Before training, the MLP's loss with small random weights must be close to ln 10 ≈ 2.303 (chance for 10 classes).

Features

Pixels divided by 16 (a constant, so nothing is fitted on validation or test). No augmentation in the learning-rate and weight-decay grid; dropout is the homework regulariser.

Candidate methods

numpy MLP 64→128→10 (ReLU, softmax cross-entropy, 9,610 parameters), plain mini-batch SGD, batch 32, 60 epochs
The r07 grid: learning rate {0.01, 0.03, 0.1, 0.3, 1.0} × weight decay {0, 1e-4, 1e-3, 1e-2}, 5 seeds each (100 runs). Small enough to run the full grid on CPU in minutes.
The same MLP with inverted dropout on the hidden layer, p in {0, 0.2, 0.5}
Homework ablation at the chosen learning rate and weight decay, at least 3 seeds per rate; dropout switched off at evaluation.
Optional: SGD with momentum, or Adam with L2 penalty against AdamW
Shows that 'weight decay' is optimiser-dependent; only if the core grid is complete.

Validation design

One stratified 60/20/20 train/validation/test split with a fixed split seed. In the grid, seeds change only initialisation and mini-batch order, so the spread you see is optimisation noise. Secondary analysis: re-run the two best cells over 5 different split seeds and compare that spread with the seed-only spread. Report validation metrics at the final epoch and at the best epoch, and say which one the selection rule uses. The test split is read once, for the chosen cell only.

Leakage checks

Constant scaling only; the test split never enters the grid; the selection rule is written into the README before the grid runs, so the choice cannot be fitted to the results after the fact.

Bias and representativeness

Two selection biases: picking the best of 20 cells overstates that cell's validation score (report the test score as the honest estimate), and a best cell on the edge of the grid means the grid was too narrow, so extend it before concluding. Data bias: 13 writers only.

Robustness checks

Seed spread per cell; split-seed spread for the two best cells; sensitivity of the conclusion to the epoch budget (results at epoch 20 against 60); divergence counted, not dropped silently.

Technical metrics

Validation log loss and accuracy (mean ± sd, n = 5 per cell), divergence count (NaN or loss above 10), generalisation gap (train minus validation accuracy), epochs to reach 95% validation accuracy, and the update-to-weight ratio per learning rate (a rough practitioner guide puts healthy values around 1e-3).

Business interpretation

Cost of noise: with the observed seed sd $\sigma$ , the number of seeds per arm needed to detect a difference $\delta$ at 5% significance and 80% power is about $n \approx 2(1.96+0.84)^2\sigma^2/\delta^2 \approx 15.7\,\sigma^2/\delta^2$ . Report $n$ for a 0.5-point accuracy difference, and the CPU minutes per grid, as the team's budget for future comparisons.

Error slices

Learning-rate rows: where training diverges or oscillates, with the first-epoch losses of diverged seeds.
Weight-decay columns: where validation loss rises again because the model underfits.
Per-class accuracy for the chosen cell (commonly confused digit pairs).
Seed-level outliers: runs more than 2 sd from their cell mean, inspected rather than averaged away.

Implementation steps

Studio 0-15 min: write the selection rule and the grid in the README; load data, split, logistic baseline.
Studio 15-40 min: MLP forward and backward with shape assertions; loss at init near 2.303; float64 gradient check; overfit 20 examples.
Studio 40-70 min: run the grid with 3 seeds per cell (reduced) and draw the mean and sd heatmaps; look at loss curves for one cell per row.
Studio 70-90 min: apply the selection rule, write the first conclusion paragraph and the decision-log entry.
Homework: complete 5 seeds per cell, the split-seed analysis, the dropout ablation, the seeds-needed calculation and one test read; final one-page conclusion.

Reproduction

Reference implementation r07 (status: implemented; recorded results in python/reference/r07_training_dynamics/RESULTS.md):

cd python && uv run python -m reference.r07_training_dynamics.run

Behaviour: numpy and scikit-learn only, no downloads, the full grid of 100 runs plus the supplementary extension and split-seed analyses in about 80 seconds on a laptop CPU (--smoke for a quick check). Writes to python/reference/r07_training_dynamics/outputs/: runs.csv (one row per run: learning rate, weight decay, seed, final and best-epoch metrics, divergence flag, per-epoch loss curves), summary.csv (mean, sd, n, failures per cell), selection.json (selection rule applied, chosen cell, test result, seeds-needed estimate), heatmap_val_logloss.png and curves.png. The written conclusion is in RESULTS.md. Unit tests cover shapes, the gradient check and the weight-decay update.

Limitations

A tiny dataset and network; the best learning rate and the size of seed noise do not transfer to larger models or other data.
Plain SGD only in the core grid; conclusions about weight decay do not carry over to Adam without the AdamW distinction.
Five seeds estimate the sd roughly; the sd of the sd is itself large at n = 5.
Random splits mix the 13 writers, so generalisation to new writers is untested.

Production and monitoring

The output is a team practice, not a deployed model: a template default (learning rate, weight decay, epochs), a minimum number of seeds for any comparison, and an experiment-log format with seed, split seed, optimiser and decay type recorded on every run. Re-run the grid when the architecture, optimiser or data size changes materially.

Interview questions

Two configurations differ by 0.3 accuracy points from single runs. What do you do before believing it?
Is an L2 penalty the same as weight decay? When does the difference matter?
Your best cell is at the largest learning rate in the grid. What does that tell you and what do you do next?
Why does a high learning rate sometimes reach a lower validation loss before diverging in other seeds?

Postmortem and alternatives

Record whether the learning rate mattered far more than weight decay (common on small, easy data), whether seed noise was larger than the differences between neighbouring cells, and whether split-seed variance exceeded initialisation variance. If the selection changed between epoch 20 and 60, the epoch budget is part of the configuration and must be reported with it. Alternatives: random search on a log scale instead of a grid, a learning-rate range test, or a schedule (warm-up and decay) instead of a constant rate.

Reference implementation

Status: executed — outputs below are from the recorded run

cd python && uv run python -m reference.r07_training_dynamics.run

r07 results - learning rate x weight decay on scikit-learn digits

Recorded run of cd python && uv run python -m reference.r07_training_dynamics.run (full mode). Numbers are copied from outputs/selection.json, outputs/summary.csv and outputs/runs.csv, written by that run. Data: bundled scikit-learn digits (1,797 images; train 1,078 / validation 359 / test 360, stratified, split seed 0, no duplicate rows). Model: numpy MLP 64-128-10 (9,610 parameters), plain mini-batch SGD (batch 32, 60 epochs, no momentum), weight decay applied in the update (identical to L2 in the loss for SGD).

Item	Value
Python / numpy / scikit-learn	3.12.13 / 2.5.3 / 1.9.1
Runtime of the recorded run	79.8 s wall, 75.3 s CPU for 175 training runs and the secondary analyses (single thread; the laptop was shared)
Grid	5 learning rates x 4 weight decays x 5 seeds = 100 runs; plus 15 supplementary extension cells x 5 seeds = 75 runs

Selection rule (written before the grid was run; see README)

Mean over seeds of the final-epoch (epoch 60) validation log loss; a cell is eligible only if no seed failed (NaN/inf or loss above 10 at an epoch end, or a final training loss of at least 2.0, i.e. never left chance); cells within one pooled seed sd of the best eligible mean are tied; ties go to the cell farthest from the divergence region, or, if nothing failed, to the smaller learning rate and then the smaller weight decay; if the chosen cell sits on the upper learning-rate or weight-decay edge the grid is extended once; the test split is read once, for the chosen cell. One disclosed amendment: the "stalled" failure criterion was added after a reduced smoke run showed a dead network that the first wording did not count; the full grid had not been run at that time.

Findings

The network passed its checks. Loss at initialisation 2.310 against ln 10 = 2.303; 20 training rows were overfit to a loss of 0.0100 in 146 full-batch SGD steps (accuracy 1.0); the float64 gradient check passes on every coordinate (largest absolute error 9.0e-11; largest relative error 1.2e-4 in W1, on a coordinate whose gradient is about 1e-7, where floating-point noise alone exceeds 1e-4 relative). Baselines on the same split: majority class 10.0% validation accuracy; multinomial logistic regression 97.8% validation and 96.7% test accuracy (log loss 0.161 / 0.162).
No cell failed in the 5 x 4 base grid; the learning rate stops working between 1.0 and 3.0. All 100 base-grid runs trained. In the supplementary extension, all 25 runs at lr 3.0 and all 25 at lr 10.0 collapsed to chance (final training loss at or above 2.0; none produced NaN or a loss above 10 at an epoch end), and 1 of the 5 seeds at lr 1.0 with weight decay 0.1 did too. Mean update-to-weight ratio in epoch 1 rose from 0.0008 (lr 0.01) to 0.096 (lr 1.0) and 0.27 (lr 3.0).
The learning rate matters far more than weight decay. Validation log loss at weight decay 0 falls from 0.304 (lr 0.01) to 0.150 (0.03), 0.0865 (0.1), 0.0639 (0.3) and 0.0522 (1.0); in the same columns, weight decay up to 1e-4 changes the loss by at most 0.003 (lr 0.3: 0.0639 vs 0.0636). The seed sd of a cell is 0.002 to 0.012 except the unstable corner (lr 1.0, wd 0.01: mean 0.442, sd 0.105).
Weight decay did not help at any learning rate in this budget. Relative to wd = 0: wd 1e-3 costs +0.007 to +0.016 (all five rates); wd 1e-2 costs +0.075 (lr 0.01), +0.10 (0.03), +0.145 (0.1), +0.168 (0.3) and +0.39 (1.0); wd 0.1 leaves the network under-fitted at 1.29 to 1.33 for every lr up to 0.3. The damage of a given decay grows with the learning rate (the update shrinks weights by lr x wd per step), which is the learning-rate x weight-decay interaction. The generalisation gap in accuracy is only 0.5 to 2.1 points for the cells that trained, so there was little overfitting for decay to remove after 60 epochs of SGD on 1,078 images.
The rule chose lr 0.3, wd 0; the lowest-mean cell is lr 1.0, wd 1e-4. Chosen cell: validation log loss 0.0639 +/- 0.0042 (sd, 5 seeds), accuracy 98.4% +/- 0.5. Lowest mean: 0.0494 +/- 0.0039. Five cells were tied (threshold 0.0733 = best 0.0494 + pooled sd 0.0239); no cell had failed, so the tie went to the smaller learning rate and then the smaller weight decay: lr 0.3, wd 0. The chosen cell is not on a grid edge, so the edge rule did not trigger and the extension cells were not used for selection.
The tie window was inflated by one unstable cell. The pooled seed sd of 0.0239 is dominated by the single cell lr 1.0, wd 0.01 (sd 0.105); without it the pooled sd would be 0.0052 and only the two lr 1.0 cells with wd 0 and 1e-4 would have been tied (post-hoc calculation, not used for selection). The difference between the chosen and lowest- mean cells, 0.0145, is about 3.6 times the seed sd of those two cells (0.0041). The rule did what it said; the pooled-over-all-cells width is a weakness of the rule, and a better rule would pool within the neighbourhood of the best cell or use the best cell's own sd. Choosing lr 0.3 also keeps a factor of 10 from the learning rate where training collapsed (3.0); lr 1.0 is only a factor of 3 below it.
The epoch budget changes the answer. Applying the same rule to epoch-20 validation losses selects lr 1.0, wd 0 (0.0568, 3 tied cells), not lr 0.3. With a short budget the largest stable rate wins; with 60 epochs the smaller rate catches up and is preferred by the tie-break. The epoch budget is part of the configuration.
Split noise is larger than seed noise. Re-running with one initialisation seed over 5 different split seeds, the validation log-loss sd is 0.028 (lr 1.0, wd 1e-4), 0.031 (lr 1.0, wd 0) and 0.036 (lr 0.3, wd 0), against seed-only sds of 0.0039, 0.0036 and 0.0042 on the fixed split: 7 to 9 times larger. Validation accuracy sd across splits is 0.65 to 0.85 points against 0.12 to 0.46 points across seeds. A difference of 0.01 log loss between neighbouring cells is well outside seed noise on one split and well inside split-to-split variation.
Test result for the chosen cell (read once): log loss 0.0936 +/- 0.0063, accuracy 97.3% +/- 0.2 (5 seeds; the re-run reproduced the grid runs exactly). That is 1.1 accuracy points below the validation accuracy of the same cell, and the logistic-regression baseline drops by 1.1 points as well (97.8% to 96.7%), so most of the gap is the test split being harder, not only selection optimism. The weakest digits are 5 (94.1%) and 3 (94.6%); each of the five most frequent confusions (1 as 8, 2 as 1, 3 as 5, 4 as 8, 5 as 4) is a single test image that all five seeds miss.
Seeds needed. With a seed sd of 0.46 accuracy points for the chosen cell, detecting a 0.5-point accuracy difference at 5% significance and 80% power needs about 13 seeds per arm (n = 15.7 sd^2 / delta^2). Using the split-seed sd of 0.85 points for the same cell instead, the formula gives about 45. No seed of any non-failed cell was more than 2 sd from its cell mean (n = 5, so this is a weak screen).

Conclusion

Use plain SGD with a learning rate of about 0.3 (batch 32, 60 epochs) and no weight decay as the template default for this network and this kind of data; lr 1.0 is slightly better on a short budget but sits next to a collapse region and is more variable (sd up to 0.1 in some cells), and weight decay above 1e-4 only hurt. Compare configurations only with at least 5 seeds on a fixed split, and about 13 seeds per arm for a 0.5-point accuracy claim; a claim that must hold across data splits needs many more, or a paired design over splits. Record the optimiser, the decay type, the epoch budget, the seed and the split seed with every run.

What this does not prove

It does not transfer beyond this network and data set: 1,797 tiny images, one hidden layer, a single constant learning rate. The best rate, the collapse threshold and the size of the noise would differ for other sizes.
Nothing is said about Adam or AdamW, where L2 and weight decay differ; only plain SGD was run. The record's dropout ablation was not run.
Five seeds give a rough sd (the sd of an sd at n = 5 is large); the "7 to 9 times" figure in finding 8 comes from 5 split seeds and a single initialisation seed.
The digits come from a small number of writers and the random split mixes them; generalisation to new writers was not tested.
The selection chose the best-looking of 20 cells with one validation split; the 0.0936 test loss is the honest estimate and it is worse than the 0.0639 the selection saw.
The tie-break and the "stalled" criterion are choices of this protocol; other defensible rules would have picked lr 1.0.

Reproduce

cd python && uv run python -m reference.r07_training_dynamics.run
cd python && uv run pytest tests/test_r07_training_dynamics.py

Builder-authored reference. It is not learner evidence and is never credited to your portfolio.

Rubric4 criteria

Criterion	Meets	Exceeds
Experimental design	Pre-registered selection rule, fixed split, at least 3 seeds per cell, divergence counted, test read once.	Separates initialisation variance from split variance and extends the grid when the best cell is on its edge.
Correctness of the training code	Shape assertions, loss at init near ln 10, gradient check, weight-decay update tested.	Tests that confirm L2-in-loss and weight-decay-in-update coincide for plain SGD at the expected rescaling.
Uncertainty reporting	Mean ± sd per cell and a heatmap; differences discussed relative to seed noise.	Seeds-needed calculation for a stated effect size and a discussion of the uncertainty of the sd itself.
Conclusion	A chosen default with an explanation of the learning-rate × weight-decay pattern.	A reusable team rule (seeds, logging fields, re-run triggers) derived from the evidence.

References6

Accounting for Variance in Machine Learning Benchmarks
Sources of variance (data sampling, initialisation, hyperparameters) and why benchmarks must randomise them.
Decoupled Weight Decay Regularization
Shows L2 regularisation and weight decay are equivalent for SGD but not for adaptive optimisers; motivates decoupled weight decay.
CS231n course notes: Neural Networks Part 3 - Learning and Evaluation (gradient checks, sanity checks, babysitting learning)
Loss at chance, overfit a tiny subset, update-to-weight ratio around 1e-3, log-scale learning-rate search.
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Dropout as a regulariser and its train/evaluation asymmetry, for the homework ablation.
On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation
Selecting among many configurations on one validation set overfits the selection; the reason for one test read.
Deep Learning
Optimisation for deep models and parameter-norm penalties.

At a glance