Classical ML, validation, metrics, interpretation, debugging · insight-0159
Count every look at the test set as a selection step and refresh the holdout when looks pile up
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A team checks the test score after each feature tweak, threshold change or model swap over several weeks.
Action
Keep a locked test set used once per decision, log every access, use cross-validation for iteration, and set aside a fresh out-of-time holdout when the test has been reused.
Reason
Each decision informed by test results fits the test set a little; many small fits add up to selection bias.
Boundary
Model development cycles, competitions, A/B pre-checks and vendor bake-offs.
Symptom or risk
The test set gradually becomes a validation set; the final number is optimistic and production performance disappoints.
Exceptions
Very large test sets tolerate a few looks with little bias, though the habit still matters.
Trade-offs
Holding out more data reduces training size; out-of-time holdouts also test drift robustness.
Pitfalls
Early stopping or threshold tuning on the test set, which are hidden test-set uses.
Lessons and concepts
Claims and their sources
Ideally the test set is kept in a vault and used only at the end of the analysis (ESL section 7.2).
Tuning hyperparameters on the test set lets knowledge of it leak into the model so metrics no longer report generalization.