LLMs, GenAI, RAG, agents, evaluation · insight-0546
Do the LoRA parameter arithmetic before planning hardware
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A team asks whether LoRA lets them fine-tune a model on the GPU they already have.
Action
Count trainable parameters as r x (d_in + d_out) per adapted matrix, sum over adapted layers, and separately budget frozen base weights, activations (which grow with batch and sequence length) and optimizer state for the adapters.
Reason
LoRA freezes pretrained weights and trains low-rank update matrices, which cuts trainable parameters and optimizer state dramatically but leaves the frozen model in memory.
Boundary
Planning PEFT runs, comparing ranks and target modules.
Symptom or risk
They assume LoRA shrinks all memory, then run out because base weights and activations still have to fit.
Exceptions
QLoRA also shrinks the frozen base by quantizing it.
Trade-offs
More adapted modules or higher rank add parameters and may add quality.
Pitfalls
Reporting the trainable-parameter percentage as the memory saving.
Example
For a 4096 x 4096 projection with r = 8, LoRA trains 8 x (4096 + 4096) = 65,536 parameters instead of 16,777,216, about 0.39 percent.
Lessons and concepts
Claims and their sources
Hu et al. describe LoRA as freezing pretrained weights and injecting trainable rank-decomposition matrices, greatly reducing trainable parameters and GPU memory requirements.