LLMs, GenAI, RAG, agents, evaluation · insight-1629

Deduplicate the fine-tuning set against itself, not only against the evaluation set

Situation
A support-ticket corpus of resolved cases becomes a fine-tuning set, and thousands of tickets are templated near-copies of the same outage.
Action
Run exact and near-duplicate detection (normalised text hashes, then MinHash or embedding similarity) inside the training set, keep one or a few representatives per cluster, and record how many rows were dropped and why.
Reason
Lee et al. found that deduplicating language-model training data reduced memorised text emission about tenfold and reached the same or better accuracy in fewer steps; that study concerns pretraining corpora, so the size of the effect on a small tuning set is for you to measure.
Boundary
Sets built from logs, tickets, emails or scraped pages.

Symptom or risk

The model over-fits the repeated cases, reproduces boilerplate verbatim, and spends epochs on rows that add no information.

Deliberately repeated canonical answers, such as legally mandated wording, kept at an intended frequency.

Aggressive thresholds delete legitimate variety; inspect samples from each dropped cluster.

Deduplicating before text normalisation, so copies that differ in whitespace or signatures survive.

Lee et al. report that models trained on deduplicated data emit memorised text about ten times less often and need fewer training steps for the same or better accuracy, on language-model pretraining datasets.
Deduplicating Training Data Makes Language Models Better