Classical ML, validation, metrics, interpretation, debugging · insight-0163

Deduplicate at the correct grain before splitting so copies do not straddle train and test

Situation
The raw table has exact or near-duplicate rows, for example re-sent records or late updates of the same event.
Action
Define the business key, resolve duplicates (keep the latest valid version) before splitting, and check after splitting that no key appears in both sets.
Reason
Duplicates across the boundary turn generalization tests into recall tests.
Boundary
Event logs, CRM exports, scraped data, image or text corpora and any data merged from several systems.

Symptom or risk

A row and its copy land on both sides of the split; the model is scored on memorized examples and validation is inflated.

Repeated identical feature vectors with different labels are genuine noise, not duplicates; keep them but note them.

Near-duplicate detection can be expensive; start with exact keys and the highest-risk sources.

Deduplicating on all columns, which misses copies that differ only by an updated revenue field.

Synthetic flight_day_raw: about 0.8% duplicate rows, half with a late revenue update, so key on flight_no and date.

Duplicates across training and test sets are among the documented forms of leakage that inflate reported performance.
Leakage and the reproducibility crisis in machine-learning-based science