Classical ML, validation, metrics, interpretation, debugging · insight-0167
Read a validation curve for one hyperparameter to locate under- and overfitting, then confirm on held-out data
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You are unsure whether to increase or decrease tree depth, regularization strength or the number of neighbours.
Action
Plot training and validation scores across a log-spaced range of the parameter; move toward the region where validation peaks, extend the grid if the peak is at an edge, and confirm the choice on an untouched test set.
Reason
Low training and validation scores indicate underfitting; a high training score with a low validation score indicates overfitting.
Boundary
Single dominant hyperparameters such as C, alpha, max_depth, min_samples_leaf or k.
Symptom or risk
Blind grid search wastes compute or lands on an edge of the grid without anyone noticing.
Exceptions
Strong interactions between hyperparameters require a joint search; one-dimensional curves can mislead then.
Trade-offs
Curves give intuition and explainability; random or Bayesian search covers many parameters more efficiently.
Pitfalls
Reporting the peak validation score as the expected performance; it was used for selection and is biased.
Lessons and concepts
Claims and their sources
Validation curves reveal under- and overfitting across a hyperparameter; a validation score used for tuning is biased and a separate test set is needed for a generalization estimate.