Neural networks, deep learning, NLP · insight-1452

Compare frozen features and fine-tuning at several training sizes before committing

Situation
A team has 600 labelled Turkish complaints and argues about frozen embeddings with a linear head versus fine-tuning.
Action
Train both at 25, 50 and 100 percent of the labels over several seeds and splits, plot score against size with intervals, choose the cheaper approach when the curves are within noise, and revisit when labels double.
Reason
Peters, Ruder and Smith compared feature extraction with fine-tuning across NLP tasks and found which is better depends on how similar the pretraining and target tasks are, so no universal winner exists.
Boundary
Small and medium labelled text sets with a pretrained encoder.

Symptom or risk

Whichever approach was tried first becomes the plan; the better option at your data size and task distance is unknown and may reverse as labels grow.

A hard latency or memory cap may already decide the question.

Repeated runs cost compute; fine-tuning needs per-run tuning.

Giving the two arms different tuning budgets; one seed per arm.

Peters, Ruder and Smith report that whether feature extraction or fine-tuning works better depends on the similarity between the pretraining task and the target task.
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks