Introduced on day 40
Transfer learning
Starting from a model pretrained on a large task and reusing it, frozen or fine-tuned, for a smaller target task.
In detail
Options run from a frozen feature extractor with a new head (cheap, low variance, may miss domain cues) through fine-tuning the last blocks to fine-tuning everything with a small learning rate (more flexible, more compute, more seed variance). It works to the degree that source and target share low-level structure, and it requires reproducing the pretrained model's input preprocessing exactly (Day 40).