Concept map: Optimizers and learning rates

Taught and used in

Deep learning and neural networks

Insights 23

• Run a learning-rate range test before tuning anything else
• Retune the learning rate whenever you change the batch size
• Use Adam to iterate fast, but test SGD with momentum before final training
• With Adam, use decoupled weight decay (AdamW), not the weight_decay L2 term
• Exclude biases and normalization scales from weight decay
• Use a gradient scaler with float16 mixed precision and compare against a float32 run
• Unfreeze gradually and use smaller learning rates for lower layers on small text datasets
• Scale the loss and the schedule correctly when you accumulate gradients

Projects 7

• Will this flight fill? A from-scratch numpy MLP vs gradient boosting on airline booking snapshots
• Catalogue thumbnails: a small CNN vs pixel baselines on Fashion-MNIST
• Day-ahead household load: a 1-D CNN or GRU against seasonal naive and lag regression
• Few labels, foreign domain: frozen ImageNet features vs a small CNN on EuroSAT land cover
• Same product, different shop: TF-IDF vs sentence embeddings for product matching on Abt-Buy
• Is it the learning rate or the seed? A repeated-seed ablation of learning rate, weight decay and dropout