LLMs, GenAI, RAG, agents, evaluation · insight-0548
Tune learning rate and target modules before raising LoRA rank
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A LoRA fine-tune underperforms and the first reaction is to raise the rank from 8 to 128.
Action
Hold rank small, sweep the learning rate, compare adapting attention projections only against attention plus MLP layers, check data quality, and raise rank last, comparing each change on the eval set.
Reason
The LoRA paper reported that very low ranks were often sufficient in its GPT-3 experiments and that, within a fixed parameter budget, adapting more attention weight types at low rank could beat adapting one type at higher rank; it did not test MLP layers, and the learning-rate sweep is a practitioner recommendation.
Boundary
LoRA and QLoRA runs on instruction or task data.
Symptom or risk
Memory and training time rise with little change in quality, because rank was not the bottleneck.
Exceptions
Large domain shifts may need higher rank or full fine-tuning; measure.
Trade-offs
Sweeps cost runs, though LoRA runs are cheaper than full fine-tunes.
Pitfalls
Copying hyperparameters from a different model size or task without a sweep.
Lessons and concepts
Claims and their sources
Hu et al. reported that, on GPT-3 175B with attention weights only, low ranks were often sufficient for LoRA and that spreading a fixed parameter budget over more weight matrices could work better than raising rank on one.