Neural networks, deep learning, NLP · insight-1429
Include Adam's epsilon in the search space before ranking optimizers
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A comparison of SGD with momentum and Adam is run with default settings, and Adam loses by a point on a tabular-style network.
Action
Give each optimizer the same trial budget, tune learning rate, momentum or betas, weight decay and epsilon on a log scale, and rank by the validation score of the best trial over several seeds.
Reason
Choi and colleagues found that optimizer rankings change with the tuning protocol and that the search space may explain past disagreements more than anything else, and that a more general optimizer should not lose to one it can approximate. Adam with a large epsilon behaves like momentum SGD, so epsilon belongs in the search.
Boundary
Choosing an optimizer for a new architecture or dataset.
Symptom or risk
The ranking reflects the search protocol, not the optimizer. Another team tuning more hyperparameters, or the same team a week later, reverses it.
Exceptions
Strict compute limits that allow one default run.
Trade-offs
Multiplies compute by the number of trials per optimizer.
Pitfalls
Tuning one optimizer with a wider range than the other.
Lessons and concepts
Claims and their sources
Choi et al. (arXiv preprint) report that optimizer rankings can change with the hyperparameter tuning protocol, that the search space may matter more than anything else in explaining conflicting comparisons, and that more general optimizers should never underperform the ones they can approximate (Adam versus momentum).
Wilson et al. report that adaptive methods can generalise worse than non-adaptive methods in some settings.