Classical ML, validation, metrics, interpretation, debugging · insight-0165

Score the full pipeline on shuffled labels to catch leakage the code review missed

Situation
A complex pipeline with encoders, selectors and resamplers reports strong performance and you cannot audit every step.
Action
Re-run the entire cross-validated pipeline with permuted labels (permutation_test_score or a manual shuffle); scores should fall to the dummy level, and any excess points to leakage.
Reason
With shuffled labels, no honest pipeline can beat chance, so a remaining gap measures information smuggled through the pipeline.
Boundary
Pipelines with target encoding, supervised feature selection, resampling or custom transformers.

Symptom or risk

A hidden step uses labels from validation rows; the leak persists even when labels carry no real signal.

Group or time structure must be preserved in the shuffle design, or the test itself becomes misleading.

Each permutation refits the pipeline; a handful of shuffles is often enough to spot a leak, though p-values need more.

Shuffling labels after encodings were computed, which tests nothing.

permutation_test_score refits the model on label-permuted data to estimate how likely the observed cross-validated score is under chance.
Cross-validation: evaluating estimator performance (scikit-learn User Guide 3.1)