Classical ML, validation, metrics, interpretation, debugging · insight-0152

Resample inside the cross-validation pipeline so validation folds keep the real class mix

Situation
You undersample or oversample the whole dataset, then cross-validate or split into train and test.
Action
Use an imbalanced-learn Pipeline (sampler then estimator) so resampling runs only during fit on each training fold, and evaluate on untouched folds.
Reason
Evaluation must reflect the population the model will meet, which includes its real prevalence and no duplicated rows.
Boundary
Any resampling: random over/undersampling, SMOTE variants, cleaning methods such as Tomek links.

Symptom or risk

Test folds are balanced artificially and may contain copies or synthetic neighbours of training rows; scores do not match deployment.

None for evaluation; resampling the test data is never representative of deployment.

Pipelines add a dependency on imbalanced-learn but remove a whole class of leakage bugs.

Using scikit-learn's Pipeline with a sampler step, which does not support samplers; use the imbalanced-learn Pipeline.

Resampling the whole dataset before splitting or cross-validation leaks information and balances the test data, giving optimistic scores; using an imbalanced-learn Pipeline delegates resampling to each training fold.
Common pitfalls and recommended practices (imbalanced-learn User Guide, section 9)