Classical ML, validation, metrics, interpretation, debugging · insight-1137
Do not treat cross-validation fold scores as independent when judging a small model gap
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Two models differ by 0.4 AUC points over ten folds and someone runs a paired t-test on the ten differences.
Action
Use repeated cross-validation and compare with the corrected resampled t-test, which adds an n_test/n_train term to the variance, or treat a gap inside the corrected interval as a tie.
Reason
The naive variance term is 1/(kr) times the sample variance; the correction multiplies it by 1/(kr) + n_test/n_train, and no variance estimator is unbiased for every data distribution.
Boundary
Model selection between near-equal candidates on small and mid-sized tabular data.
Symptom or risk
Folds share most training rows, so fold differences are correlated; the naive standard error is far too small and noise gets declared a win.
Exceptions
A gap that is large relative to fold-to-fold spread, or confirmed on a fresh test set, needs no correction.
Trade-offs
The corrected test is more cautious than the naive one; it trades missed small wins for fewer false launches.
Pitfalls
Increasing the number of repeats to shrink the naive standard error, which adds correlated samples, not information.
Example
With 10 repeats of 10-fold CV the variance term grows about 12 times (standard error about 3.5 times); the scikit-learn example moves t from 2.61 to 0.75.
Lessons and concepts
Claims and their sources
In the scikit-learn grid-search statistics example, treating fold scores as independent underestimates the variance of the test statistic; the Nadeau and Bengio corrected t-test multiplies the naive variance term by 1/(k*r) + n_test/n_train and changes t from 2.611 to 0.750.
No estimator of the variance of K-fold cross-validation is unbiased for every data distribution, and naive estimators that ignore error correlations from overlapping training sets can underestimate the variance.