Neural networks, deep learning, NLP · insight-1426draft — in review

Report the score of the tuning procedure on data it never touched

Situation
You try 60 architecture and learning-rate settings on a validation split of 500 rows and quote the best validation score as the expected performance.
Action
Keep a final test split untouched until one configuration is fixed, or nest the search inside an outer cross-validation, and report the outer-fold score together with the number of configurations tried.
Reason
Model selection can itself overfit, and tuning on the same estimate used for reporting produces optimistic bias in the reported performance.
Boundary
Any hyperparameter or architecture search on a limited validation set.

Symptom or risk

The maximum of many noisy estimates is biased upward, so the quoted figure overstates what the chosen model will do on new data.

A search over a handful of settings on a very large validation set has negligible bias.

Nested evaluation multiplies training cost; hold-out costs data.

Peeking at the test split "just once" after each change, which turns it into another validation set.

Cawley and Talbot show that over-fitting can occur in model selection itself and that tuning on the estimate also used for reporting gives selection bias in the performance evaluation.
On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation