Classical ML, validation, metrics, interpretation, debugging · insight-0158

Report a tuned model's performance from nested cross-validation, not from the search's best score

Situation
You quote GridSearchCV.best_score_ (or the best of many experiments) as the expected production performance.
Action
Wrap the search in an outer cross-validation loop (or evaluate the selected configuration once on an untouched test set) and report the outer score.
Reason
Selection is part of the procedure being evaluated; only data unseen by the selection can estimate its performance honestly.
Boundary
Hyperparameter search, model-family comparison, feature-set choice, especially on small or noisy datasets.

Symptom or risk

The best score of many configurations is optimistically biased because the same folds chose and scored the winner.

With large data and a held-out test set used once, nested CV adds little beyond cost.

Nested CV multiplies compute by the number of outer folds and reports the procedure, not one fixed model.

Expecting nested CV to output the final model; refit the search on all training data afterwards.

Non-nested CV tunes and evaluates on the same data and is optimistically biased; nested CV separates selection from assessment.
Nested versus non-nested cross-validation (scikit-learn example gallery)On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation
Using cross-validation to both tune and estimate error yields biased error estimates; nested CV gives nearly unbiased estimates.
Bias in error estimation when using cross-validation for model selection