Neural networks, deep learning, NLP · insight-1423

Shuffle the training labels once and confirm the validation score collapses to chance

Situation
A network scores surprisingly high on a new dataset, and the pipeline joins several tables and applies preprocessing before the split.
Action
Permute the label column once, before any join, encoding, feature selection or split, run the whole pipeline unchanged, and score against the permuted validation labels. Expect chance-level scores; anything better indicates a pipeline or split problem.
Reason
Networks can fit random labels on the training set, so the training score proves little, whereas a validation score above chance when every label is random cannot come from real signal and points to label information used before the split (selection, target encoding, scaling) or to a split bug.
Boundary
Any new pipeline before its first reported number, especially with many joined sources.

Symptom or risk

A leak in the features or the split makes the score high for the wrong reason, and it stays hidden until production.

Time-ordered data, where you permute labels but keep the time-aware split. Duplicate rows are not caught, since each copy gets its own random label.

One extra training run; cheap with a small model.

Shuffling only the training labels and scoring against true validation labels, which shows chance whatever the leak and tests nothing.

Zhang et al. show that standard deep networks can fit randomly labelled training data.
Understanding deep learning requires rethinking generalization