Skip to content
ML Academy

Introduced on day 15

Test-set reuse

Using the same held-out block repeatedly to choose features, models or settings, which turns it into a validation set and makes its score optimistic.

In detail

The damage is graded, not binary: one look costs little, hundreds of looks cost a lot, and nobody can see the transition. A simulation with candidates of equal true accuracy shows the best reported score rising with the number of tries while the true accuracy of the chosen model stays flat. Seal the block, count accesses, or use nested cross-validation when data are too few to seal one.