Concept map: Learning curves and early stopping

Taught and used in

Validation and metrics, Classical ML and model selection

Insights 11

• Plot a learning curve before asking for more data
• Read a validation curve for one hyperparameter to locate under- and overfitting, then confirm on held-out data
• Give early stopping its own validation split and never point it at the test set
• Know that HistGradientBoosting switches early stopping on automatically for larger datasets
• Build learning curves by number of entities, not number of rows, when rows cluster
• Search scale hyperparameters on a log scale and prefer random sampling to a grid
• Early-stop neural training on the decision metric and restore the best checkpoint
• Expect large networks to memorize noisy training labels and plan for it

Projects 11

• Passenger no-show probabilities for overbooking and reconfirmation decisions
• Predicting one-year diabetes progression with an honest error estimate: nested CV on 442 patients
• Yield-failure screening from 590 sensor signals: feature selection without fooling yourself
• Caravan-policy propensity for a direct-mail campaign: many weak signals, one held-out shot
• Will this flight fill? A from-scratch numpy MLP vs gradient boosting on airline booking snapshots
• Catalogue thumbnails: a small CNN vs pixel baselines on Fashion-MNIST