Neural networks, deep learning, NLP · insight-1418
Interpolate fine-tuned and original weights to keep robustness a fine-tune would erase
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Fine-tuning a strong pretrained model lifts accuracy on your validation split, but the model is used on inputs that look different from the fine-tuning data.
Action
Evaluate a sweep of weight interpolations between the original and fine-tuned weights on both your validation split and a shifted sample, and ship the mixing coefficient that holds up on both.
Reason
Weight-space interpolation of a zero-shot model and its fine-tuned version was reported to improve accuracy under distribution shift while keeping target accuracy, with no extra inference cost.
Boundary
Fine-tuning models with a meaningful zero-shot head, such as contrastively trained image-text models.
Symptom or risk
In-distribution accuracy rises while accuracy under a distribution shift falls below the original model's, and you only notice in production.
Exceptions
Pretrained models without a usable original head, or fine-tuning that changes the architecture.
Trade-offs
Needs a shifted evaluation sample, which many teams do not have.
Pitfalls
Interpolating weights of models fine-tuned from different starting points.
Lessons and concepts
Claims and their sources
Wortsman et al. report that averaging the weights of a zero-shot model and its fine-tuned version improves accuracy under distribution shift while keeping target-distribution accuracy, at no extra fine-tuning or inference cost.