Core day 15 of 60 · Week 3 · Part I · evaluation depth

Validation and Optimization

A no-show model posts a validation AUC of 0.98 after a week of tuning. Which rows could the model, its features and your tuning loop have seen the answers on, and what does a split that mimics deployment say?

By the end: You choose a validation scheme from what is new at deployment, show the same model scoring very differently under random, group and time-ordered splits, show the best validation score rising with the number of tuning tries while fresh-row skill stays flat, and name the leakage type behind a too-good score.

At a glance

A score is only as honest as its split: hold out what deployment will see as new, choose on one block, and report once on another.

Three roles
Training rows fit, validation rows choose, a sealed test block reports once.
Split = deployment
Hold out whole entities, the future, or both: whatever is new at prediction time.
Five leakage paths
Target, feature windows, preprocessing, groups and time; fit every step inside the fold.
Tuning inflates
The best of k scores is a maximum: it rises with tries while fresh-row skill stays flat.
Report honestly
Baseline under the same split, fold spread, number of tries, one look at the test block.

Recall · 5 min

Without notes: write the Day 2 cutoff and baseline for a no-show call-back, then the Day 9 order of operations. Then say why Day 4 expects one "significant" segment among 20 even when nothing is going on: today's tuning tries are the same arithmetic.

Open due reviews

Concept · 10 min

A validation score forecasts how the model will do after deployment. Every choice made with a block of rows (a feature, a setting, a stopping round) spends some of that block's independence. Keep three roles apart: training rows fit, validation rows choose, a sealed test block reports once, and hold out exactly what deployment will see as new.

Random splits and many tuning tries both report scores that deployment does not deliver

Predict first

The best of 50 tuning tries scores 0.80 accuracy on a 500-row test block, and every candidate truly scores about 0.75. Roughly what will the chosen model score on new data?

Text description of the figure

A simulated panel of customers over periods. A deployment selector (new customers or existing customers), a customer-effect slider and a leaked-column toggle drive a bar chart of validation R-squared for random, group, time and group-plus-time splits, with a line for the R-squared actually achieved on deployment rows and a map showing which customer-period cells are train or test. A second panel has a tuning-tries slider (1 to 200) and a test-size slider, plotting the best reported test accuracy against the true accuracy of the chosen model.

Which rows do which job Diagram

Labelled rowsordered by time
Training rowsfit parameters
Validation rowschoose candidates
Test blocksealed, one look
- once: to Report
another look
Used to choose?
- yes: to Now a validation set
Reportwith baseline, spread, tries
Now a validation setits score is a maximum

Every look that changes a decision moves a block one step to the left.

Pick the split from what deployment sees as new Map

Time-ordered, with a gap

Rolling origin; entity histories from before the cutoff stay legitimate.

Group plus time

Hold out later periods and unseen entities together.

Shuffled or stratified k-fold

Only when rows are exchangeable: no repeated entities, no drift.

Group k-fold

Whole patients, customers, stores or PNRs on one side only.

Entities can be patients, customers, stores, machines or PNRs. Histories of returning entities are legitimately known at prediction time, so holding them out would be too pessimistic.

The split painter: which cells train, which test, and what that promises Illustration

Test entities already in training: 100%. Training rows dated after the first test week: 95%.

Test entities already in training: 0%. Training rows dated after the first test week: 95%.

Test entities already in training: 100%. Training rows dated after the first test week: 0%.

Test entities already in training: 0%. Training rows dated after the first test week: 0%.

training cell
test cell
not used by this split

Random rows promise R² 0.83; the model reaches 0.14 on new entities and 0.64 on known entities in later weeks. The split closest to each deployment is the one that holds out what is new: group + time (0.33) for new entities, time (0.68) for known ones. Even the right split is one noisy estimate.

Simulation from the split lab: 40 entities x 20 weeks of history, a persistent entity effect, a slow drift and a ridge model with an entity target encoding; deployment = the next 6 weeks, for 40 new or the same 40 entities. Computed on the page.

Rows are entities, columns are weeks. Each grid counts its own leaks; the bars show the validation score each split reports against what deployment delivers.

One model, three splits Chart

Forest, shuffled 5-foldrecognizes parties and weeks0.842 AUC
Forest, group 5-fold by PNRparties held out0.692 AUC
Forest, time-ordered, gap 6mimics deployment0.619 AUC
Baseline, time-ordered, gap 60.610-0.613 under every split0.613 AUC

Computed by the application script on synthetic PNRs. The honest gain over the baseline is 0.006, not 0.23.

More tuning tries, higher best score, same skill Chart

Best validation score
Same pick on fresh rows

Application script, AUC minus 0.5: the best validation AUC rises from 0.493 to 0.638; fresh rows stay at 0.521-0.532.

Key terms: train, validation and test roles, k-fold, group k-fold, rolling origin with a gap, leakage, tuning, test-set reuse and nested CV. How to pick the split, the five leakage paths and the size of tuning optimism are in the pictures above and in the optional technical depth below.

Assumptions

Deployment is stated: today new travellers on new PNRs, scored after training; for a hospital, new patients; for a lender, next month's applicants. A different deployment needs a different split.
The test block is used once, after every choice, and is large enough that its noise is below the gaps you care about (Day 4).
Columns exist at the cutoff in production as in the table; no split detects a column recorded after the outcome.

Application and workplace decision · 10 min

Day 2 card: call a traveller back before departure; grain one traveller on a booking reference (PNR); target no-show; baseline logistic regression on lead time; metric ROC AUC (Day 16 prices it).
Data: 1,500 synthetic PNRs (about 6,000 travellers, in creation order). Each party shares an unobserved habit, each week an unobserved shock.
Block 1: the baseline and a deep forest under shuffled 5-fold, 5-fold grouped by PNR, and five time-ordered splits with a gap of 6 rows (the largest party).
Block 2: keep the best of k random 4-feature subsets of 40 weak features by a 300-row validation block, then re-score it on 3,200 fresh rows.
Predict first: under which split does the forest fall most?

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import GroupKFold, KFold, TimeSeriesSplit
rng = np.random.default_rng(15)
n = 1500                                              # booking references (PNRs), 2-6 travellers each, in creation order
size, t = rng.integers(2, 7, n), np.sort(rng.random(n))
lead, fare, party = rng.gamma(2, 15, n), rng.lognormal(5, 0.4, n), rng.normal(0, 2, n)  # party: unobserved habit of a PNR
week = rng.normal(0, 1.5, 52)[(t * 52).astype(int)]   # unobserved shock shared by every booking made in one week
logit = -2.6 + 0.9 * (lead - lead.mean()) / lead.std() + (fare < 120) + party + week
pnr = np.repeat(np.arange(n), size)
X = np.c_[lead, fare, t * 365, rng.normal(size=n)][pnr]  # lead, fare, booking day, noise
y = rng.random(len(pnr)) < 1 / (1 + np.exp(-logit[pnr]))
models = {"baseline (logistic on lead)": (LogisticRegression(), [0]),
          "deep forest": (RandomForestClassifier(150, n_jobs=-1, random_state=0), [0, 1, 2, 3])}
for name, folds in {"random 5-fold": KFold(5, shuffle=True, random_state=0).split(X),
                    "group 5-fold (by PNR)": GroupKFold(5).split(X, y, pnr),
                    "rolling time, gap 6": TimeSeriesSplit(5, gap=6).split(X)}.items():
    folds, row = list(folds), []
    for m, (model, cols) in models.items():
        a = [roc_auc_score(y[te], model.fit(X[tr][:, cols], y[tr]).predict_proba(X[te][:, cols])[:, 1]) for tr, te in folds]
        row.append(f"{m} {np.mean(a):.3f}")
    print(f"{name:22s} " + " | ".join(row))
# tuning tries: 40 weak features; keep the best of k random 4-feature subsets by a 300-row validation block
Xs = rng.normal(size=(5000, 40))
ys = rng.random(5000) < 1 / (1 + np.exp(-(0.08 * Xs.sum(axis=1) - 1.5)))
def auc(cols, rows):
    m = LogisticRegression().fit(Xs[:1500][:, cols], ys[:1500])
    return roc_auc_score(ys[rows], m.decision_function(Xs[rows][:, cols]))
subsets = [rng.choice(40, 4, replace=False) for _ in range(200)]
val = np.array([auc(c, slice(1500, 1800)) for c in subsets])
for k in (1, 3, 10, 50):
    print(f"{k:3d} tries: best validation AUC {val[:k].max():.3f} | same subset on fresh rows {auc(subsets[val[:k].argmax()], slice(1800, 5000)):.3f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 34 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

What a validation score estimates: K-fold CV worked by hand7 minThe target quantity, the K-fold estimator, a three-fold run on six numbers you can redo on paper, and the bias and variance of the estimate.

What a validation score estimates, and the K-fold estimator Formula

\begin{gathered} \mathrm{Err} = \mathbb{E}\big[L(Y, \hat f(X))\big] \\ \mathrm{CV}_K = \frac{1}{n}\sum_{k=1}^{K} \sum_{i \in F_k} L\big(y_i, \hat f^{(-k)}(x_i)\big) \end{gathered}

$\mathrm{Err}$: expected loss on new rows drawn from the deployment distribution: the quantity every split tries to estimate
$F_k$: the rows of fold k; every row is held out exactly once
$\hat f^{(-k)}$: the model refit, with every learned step, on all rows outside fold k
$L$: the loss per row (squared error, log loss); ranking metrics such as AUC are computed per fold and averaged instead

CV estimates the expected error of the fitting procedure over training sets of size n(K-1)/K, more than the error of the one model you ship (ESL, section 7.12).

Cross-validation by hand: six insurance claims, three folds Table

Fold	Training rows	Prediction (training mean)	Squared errors on the fold
Data	six claim amounts (k EUR): 2, 4, 6, 8, 10, 12	model: predict the training mean	-
Fold 1 = {2, 8}	4, 6, 10, 12	8.0	(2 - 8)^2 = 36; (8 - 8)^2 = 0
Fold 2 = {4, 10}	2, 6, 8, 12	7.0	9; 9
Fold 3 = {6, 12}	2, 4, 8, 10	6.0	0; 36
3-fold CV	mean of the six held-out squared errors	-	90 / 6 = 15.0
In-sample error	the same model fitted and scored on all six	7.0	70 / 6 = 11.7: too low
Leave-one-out	each claim predicted by the mean of the other five	mean of five	16.8
Theory	new-row error of a mean of m rows: sigma^2 (1 + 1/m); sample variance 14	-	17.5 for m = 4; 16.8 for m = 5

Hypothetical claims, verified in numpy. In-sample error reuses the rows that fitted the mean, so it is optimistic; held-out rows are not. Leave-one-out matches the theory for m = 5 exactly; 3-fold trains on 4 rows (theory 17.5), and with six rows its 15.0 is itself noisy.

Choosing K: a little bias against a lot of variance Comparison

Criterion	K = 5 or 10the usual compromise	Leave-one-out (K = n)nearly all rows each time
Each fold model trains on	80-90% of the rows	n - 1 rows
Bias of the estimate	Slightly pessimistic; most where the learning curve is still steep	Nearly unbiased
Variance of the estimate	Moderate	Can be high: the n fold models are almost identical
Cost	K fits	n fits, unless a shortcut exists

Both assume exchangeable rows: held-out rows as new as deployment rows; repeated entities and time break it (next chapters). Fold scores share training rows, and no unbiased estimator of the variance of K-fold CV exists in general (Bengio and Grandvalet, 2004), so sd / sqrt(K) understates the uncertainty.

Three roles and a protocol3 minWhat each block of rows is for, in which order to use them, and how a test block gets spent.

A validation protocol in five steps Sequence

State the deployment
What is new at prediction time: customers, patients, stores, machines, weeks, or travellers on new PNRs.
Seal the test block
Latest period and whole groups; log every access.
Fix the split rule
Group, time with a gap, or both; record seeds, row ids and library versions.
Search inside the folds
A pipeline refits every learned step per fold; count every try.
Score once
Report test score, baseline, fold spread and the number of tries.

With plenty of data a sealed latest-period block is the simplest honest report; with little data nested CV replaces it. A score without its split rule, seeds, versions and tries cannot be re-derived.

Choosing a split from deployment4 minShuffled, stratified, group and time-ordered schemes, and the question that picks between them.

Split schemes and when they are honest Table

Scheme	Holds out	Honest when	scikit-learn splitter
Shuffled k-fold	Random rows	Rows are exchangeable: no repeated entities or periods	KFold(shuffle=True)
Stratified k-fold	Random rows, class shares kept	As above, with a rare class; keeps fold counts stable	StratifiedKFold
Group k-fold	Whole entities (patient, customer, store, PNR)	The model will score entities it never saw	GroupKFold
Stratified group k-fold	Whole entities, class shares kept as far as groups allow	New entities and a rare class	StratifiedGroupKFold
Time-ordered	The latest period	The model predicts the future (Day 25)	TimeSeriesSplit

A baseline with nothing to memorize barely moves between schemes; a large move in the model's score is the signature of memorization.

Time-ordered splits: expanding, rolling and the gapadvanced4 minHow rolling-origin folds are laid out, when to slide the window, and how wide the gap must be.

Time-ordered schemes: expanding, rolling, gap Comparison

Criterion	Expanding origin	Rolling window	Gap or embargo
Training window	All rows before the cutoff; grows with each fold	A fixed recent span that slides forward	Rows dropped between the end of training and the test start
Good for	Stable patterns and small data	Drift, when recent behaviour matters most	Late-maturing labels, multi-step horizons, shocks shared by neighbours
Watch out for	Early folds train on little data; old regimes keep their weight	Less data per fit; the window length is itself a choice to validate	Make it at least the horizon plus the label delay
In scikit-learn	TimeSeriesSplit, default	TimeSeriesSplit(max_train_size=...)	TimeSeriesSplit(gap=...)

The gap combines with either window, and with groups: inside each time fold, entities seen before are allowed or excluded to match deployment. Day 25 adds horizon-wise scoring for forecasts.

Rolling origin with a gap, fold by fold (example layout) Table

Fold	Train	Gap, dropped	Test
1	Weeks 1-20	Week 21	Weeks 22-25
2	Weeks 1-25	Week 26	Weeks 27-30
3	Weeks 1-30	Week 31	Weeks 32-35
3, rolling window	Weeks 11-30	Week 31	Weeks 32-35

Each test block lies after its training rows; the gap stops labels and shared shocks bridging the cutoff. The application's gap is 6 rows, the largest party, so no PNR straddles it. Late-maturing labels (a no-show at departure, a default after 90 days) need the gap to cover that delay. TimeSeriesSplit (1.9.1) defaults: 5 splits, expanding, gap 0.

Five leakage paths and the pipeline guard4 minWhere answers sneak into training rows, which re-split exposes each, and why preprocessing belongs inside the fold.

Five leakage paths and their guards Table

Type	What leaks	Example	Guard
Target	The outcome or its consequence	A 'claim settled' flag in an insurance fraud model; gate scan status for no-shows	Availability audit at the cutoff
Feature generation	Windows that reach past the cutoff	A 30-day card-transaction count that ends after the decision date	Point-in-time joins
Preprocessing	Steps fit before the split	Scaler, selection, encoder or resampling on all rows (Day 9)	Pipeline refit inside each fold
Group	The same entity on both sides	One patient's visits, or one PNR's travellers, in train and test	Group k-fold
Temporal	Neighbouring or future periods in training	A weekly promotion shock in retail sales shared by train and test	Time-ordered split with a gap

A working five-way grouping; published taxonomies split these further. Only the first two need a timing audit; no re-split detects them.

Feature selection before or inside the folds Diagram

All rowslabels are pure noise
- to Select 20 of 2,000
All rowssame data
- to 5-fold CV
Select 20 of 2,000features, using every label
- to 5-fold CV
5-fold CV
- to Select inside each fold
5-fold CV
- to Accuracy 0.870
Select inside each foldtraining rows only, in a pipeline
- to Accuracy 0.500
Accuracy 0.870pure leakage
Accuracy 0.500the truth

Computed in the hands-on lab (block B). Scalers, encoders and resampling leak the same way, usually by less: a Pipeline passed to cross_val_score or GridSearchCV refits every step on each training fold.

Locate a leak by re-splitting one factor at a time Diagram

Score too goodshuffled folds
Drops under group folds?
- no: to Drops under time folds?
yes
Group leakagehold out whole entities
Drops under time folds?
- no: to High under every split?
yes
Temporal leakagetime-ordered split with a gap
High under every split?
- no: to Count the tries
yes
Target or window leakaudit each column's timestamp
Count the trieswhat remains is selection

A target or feature-window leak lifts every split alike, so no re-split exposes it; only an audit of each column's timestamp against the cutoff finds it.

Tuning and the winner's curse4 minWhy the best of many scores is too high, how much too high, and why a stopping round counts as a try.

How much the best of k scores overstates skill Formula

\mathbb{E}\left[\max_{j \le k} \hat s_j\right] = s + \sigma\, m_k

$s$: true score shared by k equally good candidates
$\sigma$: noise of one score on the validation block; about 0.019 for accuracy 0.75 on 500 rows
$m_k$: expected maximum of k standard normals: 1.16 at k = 5, 1.54 at 10, 2.25 at 50, 2.75 at 200

Assumes independent, normal score noise. Candidates scored on the same rows are correlated, which shrinks the optimism without removing it.

The expected-maximum table behind tuning optimism Table

Tries k	m_k: expected max of k standard normals	sqrt(2 ln k)	Optimism at 500 rows (points)	At 5,000 rows (points)
1	0	0	0	0
2	0.564	1.18	1.09	0.35
3	0.846	1.48	1.64	0.52
5	1.163	1.79	2.25	0.71
10	1.539	2.15	2.98	0.94
20	1.867	2.45	3.62	1.14
50	2.249	2.80	4.36	1.38
100	2.508	3.03	4.86	1.54
200	2.746	3.26	5.32	1.68

The maximum of k independent standard normals has CDF Phi(t)^k, so m_k is the mean of its density k phi(t) Phi(t)^(k-1), integrated numerically. Optimism = 100 x sigma x m_k with sigma = sqrt(0.75 x 0.25 / rows), 0.0194 at 500 rows. The sqrt(2 ln k) shortcut overstates every row here.

The winner of a search is mostly the luckiest draw Illustration

50 candidates sorted by their validation score (dots) with their true accuracy (bars). The winner scores 0.815; its true accuracy is 0.761. The truly best candidate (0.768) scored 0.786.

Average over 200 searches. At 50 tries the reported best is 0.798 with 500 rows, while the chosen model is truly 0.760; with 5,000 rows the gap shrinks from 0.038 to 0.007, and the less noisy block also picks a truly better model (0.770).

Simulation: candidates' true accuracies drawn around 0.75 with SD 0.01; each validation score adds binomial sampling noise (normal approximation); independent noise across candidates. Computed on the page with the split lab's engine.

Left, one search of 50 near-equal candidates; right, the average over many searches as the number of tries grows.

Optimism grows with tries and shrinks only slowly with rows Chart

Validation block, 500 rows
Validation block, 5,000 rows

Computed from the formula above with true accuracy 0.75. Ten times the rows cuts the optimism by only about 3.2 (the square root of 10), so a smaller search helps more than a bigger block. Report the number of tries next to the score.

Early stopping is tuning too Chart

Validation, at the chosen roundround 16 was picked on these rows0.759 accuracy
Sealed test, same roundthe number to report0.719 accuracy

Computed in the hands-on lab (block D). Each boosting round is one more candidate, as are features, settings and model families (Day 4: many looks, one maximum).

Nested cross-validationadvanced3 minAn outer loop for the estimate, an inner loop for the search, and what the result actually estimates.

Nested cross-validation Diagram

Outer splitk folds for assessment
Outer training folds
Inner CVthe whole search, incl. selection
- to Refit the winner
Final modelrerun the search on all rows
Refit the winneron the outer training folds
Score the outer foldthe search never saw it
per fold
Average outer scoresestimates the procedure
- then: back to Final model

The pick in each outer fold depends on the outer training rows only, so the outer test fold is fresh for the whole procedure, search and feature selection included. Cost multiplies by the outer folds. In scikit-learn, GridSearchCV's best_score_ is non-nested; cross_val_score(GridSearchCV(...), X, y) is nested.

Non-nested versus nested CV against fresh rows Illustration

Simulation: 20 standard-normal inputs, y = 0.4 x1 + 0.4 x2 + noise (a model with both true inputs reaches R² about 0.24); each candidate is a linear regression on a random set of three inputs, 30 candidates per search. Computed on the page.

Means over 6 datasets: best CV score 0.194, nested 0.168, fresh rows 0.143. The search's own best score overstates the pick by 0.052 on average; nested CV, which re-runs the search inside each outer fold, lands closer. Where every outer fold picks the same candidate, nested and non-nested coincide. Single datasets scatter widely either way.

Each column is one small simulated dataset and one 30-candidate search, scored three ways.

Tuned and reported on the same folds, or nested Chart

Best CV score of the grid searchchosen and reported on the same folds0.568 accuracy
Nested CVouter folds untouched by the search0.515 accuracy

Hands-on lab, block A, mean over 10 simulated datasets of 120 rows with a weak signal. The nested number estimates the procedure, not the one model you ship; GridSearchCV(refit=True), the default, then refits the winner on all rows.

How uncertain is a CV score?advanced3 minFold-to-fold spread, paired comparisons and the one-standard-error rule.

Fold by fold: a 0.006 gain is a coin flip Table

Time-ordered fold	Baseline AUC	Forest AUC	Forest minus baseline
1	0.627	0.608	-0.019
2	0.586	0.565	-0.021
3	0.550	0.626	+0.075
4	0.674	0.655	-0.020
5	0.627	0.640	+0.013
Mean (sd)	0.613 (0.047)	0.619 (0.035)	+0.006 (0.041)

Computed from the application script's time-ordered folds. The forest wins 2 of 5; the spread between folds is seven times the mean gain.

Reading the spread of a CV score Overview

Pair the folds
Compare models fold by fold on the same splits; the paired differences are what matter.
Correlated folds
Folds share training rows, so their scores are correlated and the naive standard error is too small.
Repeats
Repeated k-fold averages away the luck of one split; it does not fix a split that misses deployment.
One-standard-error rule
Among candidates within one standard error of the best, prefer the simplest.

Learning curves: more data or more model?2 minRead the train and validation curves to decide whether rows, features or regularization will help.

Learning curve: a deep tree Chart

Training
Validation

Hands-on lab, block C: validation rises from 0.603 to 0.671 and then flattens, while a gap of 0.17 stays open.

What the two learning curves tell you to do next Hierarchy

Learning curvestrain and validation score by rows
- wide gap, narrowingVariancemore rows or more regularization
- gap open, val flatRows buy littleregularize or find better features
- both low, closeBiasadd features or capacity, not rows

The lab's deep tree (block C) is the middle case: validation flattens near 0.67 while a gap of 0.17 stays open.

Memory anchor

Hold out what deployment will not have seen, choose on one block, report once on another.

Useful for

Designing the split and tuning protocol before any model is fit, and writing it down so reviewers can check it.
Diagnosing a validation-to-production gap by re-splitting one factor at a time.
Deciding how much tuning a comparison can afford before the reported maximum stops meaning anything.

Not suitable for

Settling whether a column is available at prediction time; no split detects that, only a timing audit does.
Strongly time-dependent forecasting with changing structure, where one expanding window is not enough (Day 25 gives rolling origins and horizon-wise scoring).

Traps

Shuffled folds that recognize the entity

Rows of one entity share traits: travellers on one PNR, visits of one patient, months of one store. A flexible model memorizes the entity from some of its rows and predicts the rest; AUC rises and production does not follow. Stratification and repeats do nothing for it.

How to avoid it

Hold out whole entities when deployment sees new ones, and compare the model with a baseline that has nothing to memorize.

One more look at the test block

Each check "just to confirm" is a selection. The score creeps up, nobody can say when the test block turned into a validation set, and the report becomes a maximum.

How to avoid it

Seal the block, log every access, and report tuning tries next to the score; use nested CV when data are too few to seal a block.

A leak that survives every split

A column recorded after the outcome lifts group, time and random splits alike, so re-splitting never exposes it, and the model collapses on the first live day.

How to avoid it

Audit each column's timestamp against the decision cutoff (Day 2) before reading any validation score.

Senior reasoning

A senior treats a validation number as the output of a protocol, not of a model. They ask what is new at prediction time, which rows influenced which choice, how many candidates produced the maximum, and what a baseline with nothing to memorize scores under the same split. They distrust implausible scores first as bug reports, re-split one factor at a time to locate the leak, and keep a sealed block so the final number is a single, pre-registered look.

Interview question

"Your validation AUC is 0.98. Why might you still not trust it?" How would you structure the answer?

Outline your answer first.

Recall cards scheduled after completion

Which rows fit, which choose and which report, and how do you pick a split scheme?
Why is the best validation score of many tries optimistic, and what fixes it?
Name five leakage types and one guard for each.

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Cross-validation: evaluating estimator performance (scikit-learn User Guide 3.1) — scikit-learn developerscore
Defines k-fold, stratified and group k-fold iterators and TimeSeriesSplit with a gap; the group iterator keeps one group's samples in one fold. (3.1.2.1.1 K-fold; 3.1.2.2.1 Stratified K-fold; 3.1.2.4.1 Group K-fold; 3.1.2.6.1 Time Series Split)
On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation — Gavin C. Cawley, Nicola L. C. Talbot, 2010core
Tuning hyperparameters on the same cross-validation estimate that is reported gives an optimistically biased score; model selection itself can overfit.
Bias in error estimation when using cross-validation for model selection — Sudhir Varma, Richard Simon, 2006optional
The minimum cross-validation error after tuning is optimistically biased; nested cross-validation gives a nearly unbiased estimate.
Nested versus non-nested cross-validation (scikit-learn example gallery) — scikit-learn developersoptional
A runnable comparison of non-nested and nested CV showing the optimistic bias of tuning and scoring on the same folds.
Common pitfalls and recommended practices (scikit-learn User Guide, section 12) — scikit-learn developerscore
Leakage through preprocessing fit on all data, the pipeline remedy, and recommendations for controlling randomness in estimators and CV splitters. (12.2 Data leakage; 12.3 Controlling randomness)
Leakage in data mining: Formulation, detection, and avoidance — Shachar Kaufman, Saharon Rosset, Claudia Perlich, Ori Stitelman, 2012optional
Leakage as information about the target that is not legitimately available at prediction time, with detection and avoidance guidance.
Leakage and the reproducibility crisis in machine-learning-based science — Sayash Kapoor, Arvind Narayanan, 2023optional
Survey of leakage in ML-based science across 17 fields with a taxonomy of eight types; the five-way grouping here is a working simplification.
Validation curves and learning curves (scikit-learn User Guide 3.5) — scikit-learn developersoptional
Validation and learning curves: more data helps while a train-validation gap stays open, not when both curves converge low; a tuned validation score is biased. (3.5.1 Validation curve; 3.5.2 Learning curve)
The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd edition) — Trevor Hastie, Robert Tibshirani, Jerome Friedman, 2009optional
The wrong and right way to do cross-validation: selecting predictors on all rows before CV gives a deceptively low error. (Chapter 7, Model Assessment and Selection; the section on the wrong and right way to do cross-validation (within 7.10 Cross-Validation))
No Unbiased Estimator of the Variance of K-Fold Cross-Validation — Yoshua Bengio, Yves Grandvalet, 2004optional
No universal unbiased estimator of the variance of K-fold cross-validation exists; naive estimators that ignore the overlap between training sets can underestimate it.
On the use of cross-validation for time series predictor evaluation — Christoph Bergmeir, José M. Benítez, 2012optional
Preview of Day 25: evaluates cross-validation variants for time series predictors against last-block holdout.

Optional sections

Deep dive · 10 min

How big is the maximum? For $k$ equally good candidates scored with independent noise of standard deviation $\sigma$ , the expected best score exceeds the true value by roughly 1.2σ at $k=5$ , 1.5σ at 10, 2.2σ at 50 and 2.7σ at 200 (expected maximum of standard normal draws; the $\sqrt{2\ln k}$ rule overstates it for small $k$ ). The optimism grows slowly with $k$ and shrinks only with the square root of the validation size. That is why a bigger validation block helps less than a smaller search.

Nested CV. The outer loop splits for assessment; inside each outer training set an inner loop runs the whole search, including any feature selection. The outer score estimates "the procedure of tuning and fitting", not one fixed model. Cost multiplies by the outer folds. With plentiful data a sealed block is cheaper and simpler.

Early stopping. The round is chosen on validation, so its score is mildly optimistic; the lab shows the size. Keep a block that took no part in stopping or tuning.

Learning curves. A training score near 1 with a lower validation score that is still rising with rows is a variance problem and more data helps. Two converged, low curves signal bias: add features or capacity, not rows.

Deployment decides the split. Returning customers' histories are legitimately known, so a time split that keeps them is right; for a new-patient or new-customer product a group split is right. Neither is "the correct" scheme.

Reproducibility. Store the split rule or row ids, seeds, library versions and the number of tries. A score without them cannot be re-derived or compared.

Hands-on lab: Nested CV, selection before CV, learning curves and early stopping · 20 min

Run the script. Block A compares a grid search's own best score with nested CV over 10 small datasets, B selects features on all rows before CV with pure-noise labels, C draws a learning curve for a deep tree, D picks a boosting round on validation and rescores it on a sealed block. Then:

Why is A's first number higher, and which number belongs in a report?
Why does B reach about 0.87 on labels that are random, and what is the fix?
In C, would 10,000 more rows close the gap? What would you change instead?
In D, how far is the validation accuracy at the chosen round above the sealed-test accuracy, and what does that say about a stopping round as a tuning try?

import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, KFold, cross_val_score, learning_curve, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier

rng = np.random.default_rng(15)

# A) tuning on the score you report: non-nested vs nested cross-validation, 10 simulated datasets (120 rows, weak signal)
grid = {"C": [0.01, 0.1, 1, 10, 100], "gamma": [1e-4, 1e-3, 1e-2, 1e-1]}
inner, outer = KFold(5, shuffle=True, random_state=1), KFold(5, shuffle=True, random_state=2)
non_nested, nested = [], []
for _ in range(10):
    X = rng.normal(size=(120, 30))
    y = (0.3 * X[:, 0] + rng.normal(size=120) > 0).astype(int)
    non_nested.append(GridSearchCV(SVC(), grid, cv=inner).fit(X, y).best_score_)
    nested.append(cross_val_score(GridSearchCV(SVC(), grid, cv=inner), X, y, cv=outer).mean())
print(f"A) best CV accuracy of the search {np.mean(non_nested):.3f} | nested CV accuracy {np.mean(nested):.3f}")

# B) feature selection on all rows before CV, with labels that are pure noise (the wrong-way example in Elements of Statistical Learning, ch. 7)
Xn = rng.normal(size=(100, 2000))
yn = rng.integers(0, 2, 100)
wrong = cross_val_score(LogisticRegression(), SelectKBest(f_classif, k=20).fit_transform(Xn, yn), yn, cv=5).mean()
right = cross_val_score(make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression()), Xn, yn, cv=5).mean()
print(f"B) select 20 of 2,000 noise features: before CV {wrong:.3f} | inside each fold {right:.3f} (truth 0.5)")

# C) learning curve: does more data close the gap?
Xc = rng.normal(size=(3000, 10))
yc = (Xc[:, :3].sum(axis=1) + rng.normal(size=3000) * 2 > 0).astype(int)
sizes, tr, va = learning_curve(DecisionTreeClassifier(max_depth=8, random_state=0), Xc, yc, cv=5, train_sizes=[0.1, 0.3, 0.6, 1.0])
for s, a, b in zip(sizes, tr.mean(axis=1), va.mean(axis=1)):
    print(f"C) {s:5d} training rows: train accuracy {a:.3f} | validation accuracy {b:.3f}")

# D) early stopping is one tuning try per round: the validation score at the chosen round is optimistic
Xa, Xte, ya, yte = train_test_split(Xc, yc, test_size=0.4, random_state=0)
Xtr, Xva, ytr, yva = train_test_split(Xa, ya, test_size=0.3, random_state=0)
gb = GradientBoostingClassifier(n_estimators=300, max_depth=3, learning_rate=0.1, random_state=0).fit(Xtr, ytr)
va_acc = [(p == yva).mean() for p in gb.staged_predict(Xva)]
te_acc = [(p == yte).mean() for p in gb.staged_predict(Xte)]
best = int(np.argmax(va_acc))
print(f"D) best round by validation: {best + 1}; validation accuracy {va_acc[best]:.3f} | sealed test at that round {te_acc[best]:.3f}")

Interview practice · 10 min

A colleague tuned 500 configurations with 5-fold CV and reports the best CV score as the expected production performance. What do you say and what do you propose?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-11 · depends on scikit-learn, numpy

1 Recall · 5 min

2 Concept · 10 min

Assumptions

3 Application and workplace decision · 10 min

4 Mini-assessment · 8 min

Technical depth · optional · 34 min in total

Useful for

Not suitable for

Traps

Senior reasoning

Interview question

Recall cards scheduled after completion

Prerequisites and related concepts

References

Optional sections

Recall · 5 min

Concept · 10 min

Application and workplace decision · 10 min

Mini-assessment · 8 min