Introduced on day 44
LoRA (low-rank adaptation)
A fine-tuning method that freezes a weight matrix W and trains two thin matrices B (d x r) and A (r x k) whose product is added to W, so only r x (d + k) numbers are learned per matrix.
In detail
A starts random and B at zero, so training begins at the base model; the update is scaled by alpha over r. After training the product can be merged into W for no added inference latency, or kept as a separate small adapter file that is swapped per task. Rank, the set of adapted matrices and the learning rate are the main tuning choices.