Feature engineering and preprocessing · insight-0127
Run feature selection inside each training fold, never on the full dataset first
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You screen hundreds of candidate features by correlation with the target or by model importance, then cross-validate the shortlist.
Action
Put the selector inside the pipeline so it is refitted on each training fold, and report the score of the whole procedure, selection included.
Reason
Selection is part of model fitting; doing it on all rows lets the validation folds choose features that happen to fit them.
Boundary
Univariate filters, recursive elimination, importance thresholds, lasso screening and manual shortlisting from exploratory plots.
Symptom or risk
The shortlist already saw validation labels; cross-validated scores look good even when every feature is noise.
Exceptions
Unsupervised screens that never look at the target (dropping constant columns) are much less dangerous, though still best kept in the pipeline.
Trade-offs
Fold-wise selection may choose different features per fold, which is honest but harder to summarize; report selection frequency.
Pitfalls
Selecting features in a notebook, forgetting, and then building a clean pipeline on the shortlist.
Lessons and concepts
Claims and their sources
Choosing predictors on all samples before cross-validation is the wrong way to cross-validate and can show low error on pure noise (ESL section 7.10.2).
In scikit-learn's pitfalls example, selecting 25 of 10,000 random features on all data gave about 0.76 accuracy instead of chance level.