Classical ML, validation, metrics, interpretation, debugging · insight-1165
Search scale hyperparameters on a log scale and prefer random sampling to a grid
- Evidence type
- empirical finding
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A grid of C in {0.1, 0.5, 1, 5, 10} and alpha in {0.001, 0.002, 0.003} is searched for a model with six hyperparameters.
Action
Draw trials at random, with loguniform distributions for C, alpha and the learning rate, set a budget in trials, and extend the range when the best value sits at an edge.
Reason
Random search is more efficient than a grid when only some hyperparameters matter, and a budget does not depend on the number of parameters; log-uniform sampling matches scale parameters.
Boundary
Tuning boosted trees, SVMs and penalized models with several hyperparameters.
Symptom or risk
The grid covers two decades of C and half a decade of alpha, which can matter over five, repeats the same few values of important axes and spends most trials on unimportant ones.
Exceptions
With one or two parameters, a small grid is simple and fine.
Trade-offs
Random draws are less reproducible by eye than a grid; fix the seed and log all trials.
Pitfalls
Using successive halving with a tiny initial resource when candidates can only be told apart on many samples.
Lessons and concepts
Claims and their sources
Bergstra and Bengio report that randomly chosen trials are more efficient than grid trials for hyper-parameter optimization, matching or beating grid-configured networks in a fraction of the computation time.
scikit-learn's randomized search lets the budget be chosen independent of the number of parameters, adding uninfluential parameters does not decrease efficiency, and a log-uniform variable is the continuous version of a log-spaced parameter.