Neural networks, deep learning, NLP · insight-0442
Unfreeze gradually and use smaller learning rates for lower layers on small text datasets
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- aviation, telecom
Summary: situation, action, reason, boundary
Situation
Fine-tuning a pretrained language model on a few hundred labelled complaint emails gives unstable results that forget general language knowledge.
Action
Train the head first, then unfreeze one block at a time from the top; give lower layers smaller learning rates than upper ones, and keep a warmup and decay schedule.
Reason
Howard and Ruder introduced gradual unfreezing and layer-specific learning rates for fine-tuning a language model, and reported matching training from scratch on 100x more data with 100 labelled examples.
Boundary
Small labelled text or image datasets on top of a pretrained model.
Symptom or risk
Large updates to all layers at once overwrite pretrained knowledge, so validation accuracy swings between epochs.
Exceptions
Parameter-efficient methods such as LoRA (Day 44) reach a similar goal by restricting what can change.
Trade-offs
More schedule hyperparameters and longer wall-clock training.
Pitfalls
Forgetting to add newly unfrozen parameters to the optimizer.
Lessons and concepts
Claims and their sources
Howard and Ruder introduce fine-tuning techniques including gradual unfreezing and discriminative (layer-wise) learning rates, and report that with 100 labelled examples ULMFiT matched training from scratch on 100x more data.