Neural networks, deep learning, NLP · insight-0415

Use Adam to iterate fast, but test SGD with momentum before final training

Situation
Your model trained with Adam is close to final and the remaining question is test accuracy, not speed of iteration.
Action
Keep Adam (or AdamW) during exploration. For the final model, run a tuned SGD-with-momentum baseline under the same split, metric and tuning budget, and keep whichever wins on validation.
Reason
Wilson et al. observed adaptive methods generalizing worse than SGD on several deep-learning models even with better training loss; the effect is task dependent, so measure it.
Boundary
Vision and tabular nets trained from scratch where final accuracy matters.

Symptom or risk

Adaptive methods can find solutions that fit training data well but generalize worse than SGD on some tasks, leaving accuracy on the table.

Transformers and large-language-model fine-tuning are usually trained with AdamW; SGD often struggles there.

SGD needs more careful learning-rate schedules and more tuning time.

Giving Adam a tuned schedule and SGD the defaults, then declaring Adam better.

Wilson et al. report that solutions found by adaptive methods generalized worse than SGD on several deep-learning tasks, even with better training performance.
The Marginal Value of Adaptive Gradient Methods in Machine Learning
Adam combines bias-corrected first and second moment estimates of the gradient.
Adam: A Method for Stochastic Optimization