Introduced on day 43
Benchmark contamination
Overlap between evaluation items and the text a model was trained, tuned or prompted on, which inflates scores without improving the capability being measured.
In detail
It occurs at several layers: training or fine-tuning data copied from the evaluation set, few-shot examples drawn from it, and public benchmark items present in a base model's pretraining corpus. Inside a project it is measurable by n-gram or near-duplicate overlap checks; for closed models it can be suspected, probed and mitigated by private held-out sets, but rarely proven absent.