Neural networks, deep learning, NLP · insight-0432
Split by source item before augmenting, never after
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- aviation, manufacturing
Summary: situation, action, reason, boundary
Situation
You multiply a small labelled image set, such as photos of damaged baggage or defect images, with crops and flips, then split the enlarged set randomly.
Action
Assign each original item (and all items from the same claim, part or session) to one split first, then augment only the training split. Use a group-aware splitter keyed on the source ID.
Reason
Validation must mimic deployment, where every input is a new item; copies across the boundary are a form of leakage between training and evaluation.
Boundary
Images, audio, text paraphrases, oversampled minority rows and any generated variants.
Symptom or risk
Augmented copies of the same original land in both train and validation, so validation measures recognition of near-copies and accuracy drops sharply on new items.
Exceptions
Test-time augmentation applied to evaluation inputs is fine, because it never crosses splits.
Trade-offs
Group splits give smaller effective validation sets and noisier estimates.
Pitfalls
Near-duplicates already present in raw data, such as several photos of the same bag, need grouping too.
Lessons and concepts
Claims and their sources
Kapoor and Narayanan's survey documents leakage across many ML-based studies, including duplicates and non-independence between training and test data.
scikit-learn provides group-aware cross-validation splitters that keep all samples of a group in the same fold.