Neural networks, deep learning, NLP · insight-0453
Deduplicate near-identical texts across splits before trusting a text score
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, Applied AI, ML engineer
- Industries
- aviation, saas
Summary: situation, action, reason, boundary
Situation
A complaint or FAQ dataset contains forwarded emails, copy-pasted templates and resubmitted tickets, and you split it randomly.
Action
Normalize text (case, whitespace, signatures), detect near-duplicates with hashing such as MinHash or embedding similarity, and keep each cluster in one split. Also report the score on the deduplicated test subset.
Reason
Published NLP benchmarks contain substantial train-test overlap, and models score much worse on items that cannot be memorized.
Boundary
Support tickets, reviews, QA pairs, code and any scraped or user-generated text.
Symptom or risk
Near-duplicates sit on both sides of the split, so the model is rewarded for memorization and the test score overstates performance on genuinely new text.
Exceptions
When production traffic really repeats the same messages, report both the overlap and the novel-text scores.
Trade-offs
Aggressive deduplication can drop legitimate variants and shrink rare classes.
Pitfalls
Exact-match deduplication only, which misses paraphrases and template fills.
Lessons and concepts
Claims and their sources
Lewis et al. found 30% of test questions in three open-domain QA datasets had a near-duplicate paraphrase in training, and models performed dramatically worse on questions that could not be memorized.
Lee et al. found train-test overlap affecting over 4% of the validation set of standard language-modeling datasets.
Elangovan et al. show train-test overlap in public NLP datasets inflates reported performance.