LLMs, GenAI, RAG, agents, evaluation · insight-0538

Deduplicate fine-tuning data against the eval set before training

Situation
Fine-tuning data is exported from the same ticket archive that supplied the eval set.
Action
Before training, remove exact and near-duplicate matches between training and eval items (normalised text, n-gram overlap or embedding similarity), split by time or customer where possible, and log what was removed.
Reason
Benchmark contamination studies show test data leaking into training inflates scores; the same mechanism applies to your own fine-tuning data.
Boundary
Fine-tuning, few-shot pools and retrieval indexes built from the same source as the eval set.

Symptom or risk

Some eval tickets or near-copies end up in training, scores look excellent, and production quality is lower than reported.

Templates that legitimately repeat, such as standard notices, need a judgment call on what counts as a duplicate.

Deduplication can shrink the training set and removes some realistic repetition.

Checking only exact duplicates when tickets differ by a name or date.

Sainz et al. argue that benchmark test data present in pretraining corpora inflates scores and that contamination must be measured for each benchmark.
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark