LLMs, GenAI, RAG, agents, evaluation · insight-0554

Treat public benchmark scores as possibly contaminated and keep a private held-out set

Situation
A vendor's model posts a high score on a well-known public benchmark, and the team expects similar quality on its own tasks.
Action
Keep a private eval set that has never been published or sent to vendors for training, add items created after model cutoffs where possible, and refresh a share of items periodically.
Reason
Research shows benchmark test data can leak into pretraining corpora and inflate scores, and that contamination can be detected only with dedicated checks.
Boundary
Any use of public benchmarks for model selection or progress claims.

Symptom or risk

Part of the score reflects benchmark items seen during pretraining, so expected quality does not appear in production.

Benchmarks with demonstrated low contamination remain useful for shortlisting.

Private sets cost effort to build and refresh, and they are not comparable across organisations.

Publishing your eval set in a blog post or pasting it into tools that may train on it.

Sainz et al. argue that benchmark test data present in pretraining corpora inflates scores and that contamination must be measured per benchmark.
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Golchin and Surdeanu propose prompting-based methods to detect instance- and partition-level contamination in LLMs.
Time Travel in LLMs: Tracing Data Contamination in Large Language Models