Classical ML, validation, metrics, interpretation, debugging · insight-0164

Run a fixed leakage playbook when validation looks too good

Situation
A first model beats the incumbent by a wide margin, or one feature dominates importance.
Action
Check in order: when each top feature is written relative to the prediction time; single-feature models for suspiciously strong predictors; entity and duplicate overlap across splits; preprocessing fitted outside folds; and an out-of-time holdout.
Reason
Most leakage comes from a small set of mechanisms (target proxies, temporal misalignment, shared entities, preprocessing on all data), so a checklist finds it faster than intuition.
Boundary
Any supervised model before stakeholder review, especially with operational data sources.

Symptom or risk

Leakage produces exactly this pattern; shipping it wastes a launch and damages trust in the team.

Some problems are genuinely easy; the playbook confirms that instead of assuming it.

The checks take a day or two; a leaked launch costs far more.

Stopping after the first fix, since leaks often come in pairs.

Leakage can be formulated and detected systematically, for example by examining feature availability relative to prediction time.
Leakage in data mining: Formulation, detection, and avoidance
A taxonomy of leakage types includes lack of train-test separation, preprocessing on all data, duplicates, temporal leakage and non-independence.
Leakage and the reproducibility crisis in machine-learning-based science