Classical ML, validation, metrics, interpretation, debugging · insight-1162
Choose the simplest regularization setting within one standard error of the best
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Cross-validation picks the lowest-error alpha from a path of 100 values and the error curve is flat across a wide stretch.
Action
Plot the cross-validated error with its standard error along the path and choose the most regularized alpha whose error is within one standard error of the minimum.
Reason
Models inside that band are statistically hard to tell apart, so the simpler one costs little accuracy and is more stable.
Boundary
Lasso, ridge, elastic net and any one-dimensional complexity path: tree size, number of components.
Symptom or risk
The minimum is partly noise, so you select a less regularized model with more nonzero coefficients than the data justify, and its selected set flips between reruns.
Exceptions
When predictive accuracy is the only goal and the error curve is sharply curved, the minimum is a fine choice.
Trade-offs
A small, usually tolerable accuracy loss for sparsity and stability.
Pitfalls
Applying the rule when candidates are not ordered by complexity.
Lessons and concepts
Claims and their sources
The one-standard-error rule selects the simplest model whose performance is within one standard error of the best model's, to avoid overfitting; scikit-learn demonstrates it with a custom refit callable.