Introduced on day 15
Nested cross-validation
Cross-validation with a second, inner loop that runs the whole tuning search inside each outer training set, so outer folds assess the procedure on rows the search never saw.
In detail
The inner loop picks hyperparameters (and any feature selection); the outer loop scores the winner. The outer mean estimates the performance of "tune then fit", not of one fixed configuration. Cost multiplies by the number of outer folds. Compare with the best score of a single search, which is optimistically biased (Cawley and Talbot, 2010; Varma and Simon, 2006).