Classical ML, validation, metrics, interpretation, debugging · insight-0160
Compare candidate models on identical folds and read per-fold differences before declaring a winner
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Two models differ by a few thousandths in mean cross-validated score.
Action
Fix the splitter with an integer random_state, score both models on the same folds, inspect the paired per-fold differences, and prefer the simpler model when the gap is not consistent.
Reason
Paired comparisons remove variation due to which rows landed in which fold, so small but consistent gains become visible and noise does not.
Boundary
Model selection, feature ablations and hyperparameter comparisons.
Symptom or risk
The difference is within fold-to-fold noise; the team ships the more complex model for an imaginary gain.
Exceptions
Fold scores are correlated, so formal tests on them are approximate; use repeated CV for more stable estimates.
Trade-offs
Repeated CV costs compute; a rule of preferring simpler models within one standard error is cheap and robust.
Pitfalls
Passing a RandomState instance to the splitter, which yields different folds on each split call and invalidates fold-by-fold comparisons.
Lessons and concepts
Claims and their sources
Passing a RandomState instance to a CV splitter produces different folds on each call, which makes fold-by-fold comparisons between estimators invalid; use an integer seed.