LLMs, GenAI, RAG, agents, evaluation · insight-0547
Expect LoRA to learn less and forget less than full fine-tuning
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A team uses LoRA to teach a model a large new skill domain, such as a proprietary programming language, and gains are small.
Action
Treat the method as a trade-off and measure both sides, target-domain eval and a general-capability canary set, for LoRA at a few ranks and for full fine-tuning if affordable.
Reason
A controlled study found LoRA substantially underperformed full fine-tuning on code and math target domains but better preserved performance outside them.
Boundary
Large domain shifts, continued pretraining, deciding between PEFT and full fine-tuning.
Symptom or risk
They conclude the data is bad, or they switch to full fine-tuning without measuring what the model forgets elsewhere.
Exceptions
For narrow format and style tasks, LoRA is usually close enough, which your eval set will confirm.
Trade-offs
Full fine-tuning costs more memory and risks more forgetting; LoRA may cap target gains.
Pitfalls
Evaluating only the target task and missing regressions in general skills.
Lessons and concepts
Claims and their sources
Biderman et al. found LoRA substantially underperformed full fine-tuning on code and math target domains while better maintaining base-model performance on tasks outside them.