Neural networks, deep learning, NLP · insight-0441
Start transfer learning with a linear probe, then fine-tune if it pays
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- ML engineer, Data scientist, Applied AI
- Industries
- healthcare, insurance
Summary: situation, action, reason, boundary
Situation
You have a few thousand labelled examples and a pretrained backbone, and must decide whether to freeze it or fine-tune it.
Action
First train only a new head on frozen features. Then fine-tune the whole network starting from that trained head with a small learning rate, and keep it only if it beats the probe on an out-of-time or shifted validation set.
Reason
Early layers transfer broadly while later layers are task specific; Kumar et al. show fine-tuning from a random head can beat linear probing in-distribution yet lose out-of-distribution, and initializing from the probed head helps.
Boundary
Vision and text classifiers with limited labels or expected shift.
Symptom or risk
Full fine-tuning with a randomly initialized head can distort useful pretrained features, overfit small data and generalize worse under distribution shift.
Exceptions
Large in-domain datasets usually justify full fine-tuning from the start.
Trade-offs
Two stages cost more runs; the probe alone is cheapest to serve and maintain.
Pitfalls
Judging the choice on in-distribution validation only.
Lessons and concepts
Claims and their sources
Yosinski et al. found early-layer features are general while later layers are task specific, and transferability falls with task distance.
Kumar et al. report full fine-tuning averaged 2% higher in-distribution but 7% lower out-of-distribution accuracy than linear probing across 10 shift datasets.