Core day 44 of 60 · Week 9 · Part II · orientation depth

Parameter-Efficient Fine-Tuning

Your Day 43 evaluation set shows format and behaviour failures that a good prompt does not fix, and you have one GPU. What does LoRA change, how many numbers does it train compared with the full matrix, and what do the memory savings cost in quality?

By the end: You compute LoRA trainable parameters as r x (d + k) for a given matrix and model, estimate approximate training memory for full fine-tuning, LoRA and QLoRA, and say how an adapter is compared with the Day 43 baseline before adoption.

At a glance

LoRA freezes the base and trains a low-rank update, cutting trainable weights and memory by orders of magnitude; whether it helps is still measured on the frozen set.

Freeze and add
W stays frozen; only the thin matrices A and B learn, and B starts at zero.
Count
r(d + k) per matrix: rank 8 on 4096 x 4096 is 0.39% of the matrix.
Memory
Full Adam fine-tuning is about 16 bytes per weight; LoRA pays mainly for the frozen base.
QLoRA
A 4-bit frozen base cuts memory again; measure the quality cost on your task.
Still an experiment
Beat the best prompt on the same items, check regressions, test the served artefact.

Recall · 5 min

No reviews due? Answer from memory. (1) Which missing capability makes fine-tuning, rather than retrieval, the right lever (Day 43)? (2) What must exist before any adaptation run (Day 43)? (3) What does a rough parameter count of $12\,L\,d^2$ tell you about a model with 32 layers and width 4096 (Day 42)?

Open due reviews

Concept · 10 min

Full fine-tuning updates every weight and needs a gradient and optimizer state for each: many times the memory of running the model. LoRA bets that the needed change is low-rank: freeze $W$ and learn two thin matrices whose product is the update. You train a fraction of a percent of the weights, ship a megabyte-scale file, and merge it with no added latency.

LoRA trains r x (d + k) numbers per matrix, a fraction of a percent of the full matrix at small ranks

Predict first

Doubling the rank r on a fixed d x k matrix changes the LoRA trainable count for that matrix how?

Text description of the figure

Sliders for d, k, rank, adapted matrices per layer and number of layers, with a switch between a 12 L d^2 base-size estimate and a typed base size. Readouts give trainable parameters for the adapter and for the full model, their ratios, and approximate model-state memory for full fine-tuning, LoRA on a 16-bit base and LoRA on a 4-bit base, all before activations. Memory is in GB (10^9 bytes) and the 4-bit base ignores quantization constants, so at the defaults it shows 103, 12.95 and 3.29 GB where the application code prints 96.0, 12.1 and 3.2 GiB. Raising the rank grows the count linearly.

Four ideas to hold on to Overview

Freeze
The pretrained weights do not change and need no optimizer state.
Low-rank update
Two thin matrices B and A; their product is the change to W.
Quantize the base
Store the frozen weights in 4 bits; train adapters through them.
Prove it
Same frozen set, same items, paired with the best prompt.

The LoRA forward pass for one weight matrix Diagram

Input xk numbers
- to Frozen W
Ar x k, random init
- r numbers: to B
Frozen Wd x k, no gradients
- to h = Wx + update
Bd x r, zero init
Scale by alpha / r
h = Wx + updated numbers

Because B starts at zero, training starts exactly at the base model (Hu 2022). After training, BA can be merged into W, so serving costs nothing extra.

W + BA, cell by cell, and what a rank-r update can reach Illustration

W has 144 numbers and stays frozen. B (12 x 2) and A (2 x 12) hold 48: r(d + k) = 2 x (12 + 12). Their product has every cell of W but only rank 2: its rows are blends of A's 2 rows, which is why the update looks striped.

Share of a target update that the best rank-r matrix captures. A narrow change is 99% captured at r = 2; a broad change needs r = 7 of 12 to reach 90%. Rank caps what an adapter can express, whatever the training budget.

Seeded random matrices (12 x 12, rank 2 shown); singular values computed with the academy’s eigen-solver. The two target updates are constructed, not measured from a real model; Hu et al. 2022 hypothesise that real adaptation updates are closer to the narrow kind.

Left, a frozen 12 x 12 matrix and a rank-2 update. Right, the share of two different target changes that the best rank-r matrix captures.

Adaptation methods compared Comparison

Criterion	Full fine-tuning	LoRA	QLoRA	Bottleneck adapterHoulsby 2019
What trains	Every weight	Low-rank B and A on chosen matrices	LoRA adapters through a 4-bit base	Small modules inserted in each layer
Memory (model state)	About 16 bytes per weight with Adam, mixed precision	2 bytes per frozen weight + adapter state	About 0.52 bytes per frozen weight + adapter	Frozen base + module state
Inference cost	Unchanged	None once merged	Depends on how the base is served	Extra computation at inference
Artefact	A full model copy per task	Megabyte-scale adapter; many per base	Adapter; merging depends on backend	Module weights
Main risk	Memory; forgetting base behaviour	Too low rank underfits a large shift	Quantization error; measure on tasks	Added latency

Memory figures are approximate lower bounds before activations, KV cache and overhead.

Where the training memory goes Illustration

trained weights
gradients
fp32 master copy
Adam moments
frozen base

Arithmetic on a hypothetical configuration with the lesson’s byte counts (Rajbhandari et al. 2020 for Adam; Dettmers et al. 2023 for 4-bit storage); the totals match the application code. Activations, KV cache, temporary buffers and fragmentation come on top.

Full fine-tuning pays 16 bytes for each of 6.4 billion weights, and the optimizer state, not the weights, is most of it. LoRA trains 4.19 million weights, whose full training state is only 64 MiB; what remains is the frozen base, which is why quantizing the base, not lowering the rank, is what moves the budget.

The application's three plans split into frozen base, trained weights, gradients, master copy and optimizer moments.

Update: $\Delta W = BA$ with $B$ ( $d\times r$ ) and $A$ ( $r\times k$ ); trainable count $r(d+k)$ instead of $dk$ .
Quantizing the frozen base (QLoRA, 4-bit) moves the memory budget further; an Adapter (parameter-efficient) (Houlsby) is the older, inserted alternative.
LoRA is a training method, not a result: the adapter must beat the Day 43 best prompt on the frozen set.

Assumptions

Memory figures are rules of thumb that exclude activations, KV cache and overhead; measure on your hardware.
The configuration is hypothetical (32 layers, width 4096); real models use grouped-query attention and gated feed-forward layers that change shapes.
The needed update is low-rank enough; tasks that require large knowledge changes underfit at small rank.

Application and workplace decision · 10 min

Hypothetical 6.4-billion-weight model (32 layers, width 4096); one GPU assumed to have 24 GiB; the Day 43 format and behaviour failures.
The script counts adapter parameters for four ranks on the query and value projections, then estimates memory before activations for full fine-tuning, LoRA on a 16-bit base and QLoRA.
Predict first: which plan exceeds 24 GiB before activations are even counted?

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

GIB = 2**30
d = k = 4096  # one attention projection in a hypothetical 32-layer, width-4096 model
layers, matrices = 32, 2  # adapt the query and value projections of every layer
base = 12 * layers * d * d  # about 6.4e9 weights (the Day 42 rule of thumb, approximate)

print(f"full matrix: {d * k:,} weights; model total about {base:,}")
for r in (4, 8, 16, 64):
    per = r * (d + k)  # B is d x r, A is r x k
    total = per * layers * matrices
    print(f"r={r:<2} per matrix {per:>7,} ({per / (d * k):.2%})  all adapted {total:>10,} ({total / base:.3%} of base)")

r = 8
adapter = r * (d + k) * layers * matrices
full = base * 16 / GIB  # fp16 weights+grads (4 B) + fp32 master copy and Adam moments (12 B), before activations
lora = (base * 2 + adapter * 16) / GIB  # frozen 16-bit base + training state for the adapter only
qlora = (base * 4.127 / 8 + adapter * 16) / GIB  # 4-bit base plus about 0.127 bit/weight of quantization constants
print(f"approx. GiB before activations: full fine-tune {full:.1f}, LoRA r=8 on 16-bit base {lora:.1f}, QLoRA r=8 {qlora:.1f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 32 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The LoRA update, defined and counted by hand4 minThe forward pass, the parameter-count formula and the application's count worked line by line.

The LoRA update Formula

h = W x + \frac{\alpha}{r} B A x

$W$: frozen pretrained matrix, d x k
$B, A$: trainable, d x r and r x k; B starts at zero
$r$: rank; caps the rank of the update
$\alpha$: scaling constant; alpha over r keeps step sizes comparable across ranks

Trainable weights Formula

N_{LoRA} = L \cdot m \cdot r\,(d + k) \quad \text{vs} \quad d\,k \text{ per matrix}

$L$: layers
$m$: adapted matrices per layer (2 for query and value)
$r(d+k)$: 4096 x 4096 at r = 8: 65,536 against 16,777,216

The application's adapter count, by hand Sequence

One matrix
B is 4096 x 8 = 32,768 numbers and A is 8 x 4096 = 32,768: r(d + k) = 8 x 8,192 = 65,536.
Against the full matrix
4096 x 4096 = 16,777,216, so the pair is 65,536 / 16,777,216 = 1/256 = 0.39% of it.
Per layer
Query and value projections: 2 x 65,536 = 131,072.
Whole model
32 layers x 131,072 = 4,194,304 trainable numbers.
Share of base
Base about 12 x 32 x 4096^2 = 6,442,450,944, so 4,194,304 / 6.44e9 = 0.065%.

Each line matches the application output. The ratio r(d + k) / dk = 2r/d for a square matrix, so it falls as models get wider.

Why alpha and r move together Formula

\Delta W = \frac{\alpha}{r}\, B A, \qquad \Delta W_{\text{rsLoRA}} = \frac{\alpha}{\sqrt r}\, B A

$\alpha / r$: LoRA scaling: r 8 and alpha 16 give 2.0; r 32 with the same alpha gives 0.5
$\alpha / \sqrt r$: rank-stabilised variant (PEFT use_rslora); shrinks more slowly as r grows
$B A$: the trained low-rank product; B = 0 at start, so the update starts at zero

Changing r without changing alpha also changes the effective learning rate of the adapter, so compare ranks at a re-tuned learning rate.

Why the update has rank at most r, by handadvanced4 minThe rank argument for W + BA, a 4 x 3 rank-1 update computed on paper, and the error floor it implies.

Why the update has rank at most r Formula

\Delta W = B A, \quad \Delta W_{:,j} = \sum_{i=1}^{r} A_{ij}\, B_{:,i} \;\Rightarrow\; \mathrm{rank}(\Delta W) \le \min(\mathrm{rank}\,B, \mathrm{rank}\,A) \le r

$\Delta W_{:,j}$: column j of the update, one of k columns, each d long
$B_{:,i}$: column i of B: only r such columns exist
$A_{ij}$: how much of B's column i goes into update column j
$\le r$: every column of the update lies in the span of B's r columns, so at most r independent directions can change

The cap is a property of the parameterisation, not of training. No number of steps lets a rank-8 adapter express a change that needs 50 independent directions.

A rank-1 update by hand: d = 4, k = 3, r = 1 Table

Row of B (d x 1)	B_i x A, with A = (2, 0, 1)	What it shows
B_1 = 1	(2, 0, 1)	Row 1 is A itself
B_2 = 2	(4, 0, 2)	Row 2 is 2 x A
B_3 = 0	(0, 0, 0)	Row 3 is unchanged
B_4 = 1	(2, 0, 1)	Row 4 equals row 1
Count	12 entries changed by 4 + 3 = 7 trained numbers	r(d + k) = 1 x (4 + 3) = 7 against dk = 12

Every row is a multiple of A and every column a multiple of B, so the 4 x 3 update has rank 1 (numpy.linalg.matrix_rank confirms). At 4096 x 4096 the same structure with r = 8 trains 1/256 of the entries, a 99.6% saving.

The rank floor: the error no amount of training removes Formula

\min_{\mathrm{rank}(BA) \le r} \lVert D - BA \rVert_F^2 = \sum_{i > r} \sigma_i(D)^2

$D$: the change the task actually needs in this matrix
$\sigma_i(D)$: singular values of D, largest first
$\sum_{i>r}$: the dropped singular values squared: the best a rank-r adapter can do is the truncated SVD of D (Eckart-Young, Day 41)

Diagnostic: a validation loss that plateaus at the same value across learning rates while a higher rank goes lower is this floor, not an optimisation problem. The training demo below shows each rank stopping at its floor.

How the adapter learns: gradients and the zero startadvanced4 minThe gradients of A and B derived from the chain rule, two steps computed by hand, and why B starts at zero.

How the two thin matrices get their gradients Formula

h = W x + s\, B A x, \quad \frac{\partial \mathcal{L}}{\partial B} = s\, g\, (A x)^{\top}, \quad \frac{\partial \mathcal{L}}{\partial A} = s\, B^{\top} g\, x^{\top}

$g = \partial \mathcal{L} / \partial h$: the gradient arriving from the layers above, d numbers (backprop, Day 33)
$s = \alpha / r$: the LoRA scale
$W$: frozen: no gradient is stored for it, which is where the memory saving comes from
$B = 0$: at the start the gradient of A is exactly zero; only B moves on the first step

Both are outer products of vectors already computed. Backprop still multiplies g by W transposed to reach lower layers, so compute per step falls far less than memory: W needs no stored gradient or optimizer state.

The first two steps, by hand (d = k = 2, r = 1, s = 1) Sequence

Setup
x = (1, 2), A = (0.5, -0.5), B = (0, 0), upstream gradient g = (1, -1).
Forward
Ax = 0.5 - 1.0 = -0.5, and BAx = 0: the output is exactly the frozen model's.
Gradient of B
s g (Ax) = (1, -1) x (-0.5) = (-0.5, 0.5): nonzero, because A is random.
Gradient of A
s (B^T g) x^T = 0, because B = 0: A cannot move yet.
Step 1
Learning rate 0.1: B becomes (0.05, -0.05).
Step 2
Now B^T g = 0.05 + 0.05 = 0.1, so A's gradient is 0.1 x (1, 2) = (0.1, 0.2) and both matrices learn.
Why this init
B = 0 gives an exact start at the base model; random A gives B a direction. Both zero would never move.

Checked in numpy, including a finite-difference check of the gradient of B. Hu et al. 2022 use this initialisation (random A, zero B).

LoRA training, step by step, against its rank floor Illustration

Every run starts at relative error 1.00: B = 0 means the adapter adds nothing at step 0. Each rank stops at its floor (dashed), the best any rank-r update can do: rank 1 ends at 0.22, rank 4 at 0.00; the full update reaches 0.00.

Every run starts at relative error 1.00: B = 0 means the adapter adds nothing at step 0. Each rank stops at its floor (dashed), the best any rank-r update can do: rank 1 ends at 0.72, rank 4 at 0.26; the full update reaches 0.00.

LoRA rank 1
LoRA rank 2
LoRA rank 4
full update (dashed)

Seeded 12 x 12 linear layer with whitened inputs; plain gradient descent for 200 steps, alpha = 4, A Gaussian, B zero. Error is ||s B A − dW*||² relative to ||dW*||². A toy, not a language model: it isolates what rank does.

Gradient descent on a toy layer from the LoRA initialisation. Each rank converges to the best error a rank-r update allows.

Placement and rank3 minWhich matrices to adapt and how the trainable count moves with each choice.

Where you place adapters matters as much as rank (optional lab) Chart

q, v at r=84.19 M weights
q, v at r=3216.78 M weights
All attention at r=88.39 M weights
All attention at r=3233.55 M weights
Attention + MLP at r=818.87 M weights
Attention + MLP at r=3275.5 M weights

Same hypothetical 32-layer model. Even the largest option trains 1.17% of the base. Tune learning rate and data quality before raising rank; rank is rarely the first bottleneck.

Trainable share grows linearly with rank (query and value, 32 layers) Chart

Application output (r 4 0.033%, r 64 0.52%). Doubling r doubles the count; adapting more matrices multiplies it.

LoraConfig defaults in Hugging Face PEFT (source, as read 2026-10-12) Table

Field	Default	What an expert checks
r	8	Rank: tune after learning rate and data
lora_alpha	8	Scale is alpha / r = 1.0 at the defaults; doubling r alone halves it
lora_dropout	0.0	Some dropout can help on small data; validate it
target_modules	None (per-architecture default)	Print which matrices were adapted; placement changes the count
init_lora_weights	True: B starts at zero	Training starts exactly at the base model
use_rslora	False	True scales by alpha / sqrt(r), keeping updates stable at higher rank

Read from the PEFT LoraConfig source on GitHub; defaults change between releases, so pin the version and log the full config.

Memory accounting, derived and computed by hand5 minWhere 16 bytes per weight come from, where QLoRA's 4.127 bits come from, and the application's three figures recomputed on paper.

Bytes per weight before activations: training state, not rank, sets the bill Chart

Full fine-tune (Adam, mixed prec.)2 weights + 2 grads + 4 fp32 master + 8 Adam moments16 bytes
LoRA: frozen 16-bit weightonly the adapter's weights pay the 16 B2 bytes
QLoRA: frozen 4-bit NF4 weight4 bits + about 0.127 bits of constants0.516 bytes

Rajbhandari et al. 2020 (16 B accounting) and Dettmers et al. 2023 (double quantization). In both adapter plans the 16 B applies to millions of adapter weights, while the frozen base costs its storage for billions.

Where QLoRA's 4.127 bits per frozen weight come from Bridge

4-bit NormalFloat code4 bits
32-bit absmax scale per 64 weights32 / 64 = 0.5 extra bits per weight+0.5 bits
Scales re-stored in 8 bitsdouble quantization: 8 / 64 = 0.125 bits remain−0.375 bits
32-bit constant per 256 scales32 / (64 x 256) = 0.00195, about 0.002+0.002 bits
Bits per frozen weight4.127 bits

Dettmers et al. 2023, Section 3 (double quantization), recomputed in Python. 4.127 / 8 = 0.516 bytes per frozen weight; the 16-bit LoRA matrices and their optimizer state are counted separately.

The application's three memory figures, by hand (before activations) Table

Plan	Arithmetic	Result
Full fine-tune	6,442,450,944 weights x 16 B = 1.031e11 B; divide by 2^30	96.0 GiB
LoRA, 16-bit base	base x 2 B = 12.00 GiB, plus 4,194,304 adapter weights x 16 B = 0.06 GiB	12.1 GiB
QLoRA	base x 0.516 B = 3.10 GiB, plus the same 0.06 GiB of adapter state	3.2 GiB
Still missing	Activations (batch x sequence x width x layers), KV cache, framework overhead	Measure on a short run

Recomputed in Python; matches the application output. The adapter's share is the same 0.06 GiB in both plans, so rank barely moves the total. Activations come on top: measure peak memory on a short real run; gradient checkpointing trades about one extra forward pass for most of it (Chen et al. 2016).

QLoRA and quantizationadvanced3 min4-bit NormalFloat, double quantization and why perplexity is not the test.

What QLoRA adds Sequence

4-bit NormalFloat
A 4-bit data type for the frozen base weights.
Double quantization
Quantizes the quantization constants too: about 0.5 down to 0.127 bits per weight.
16-bit adapters
LoRA matrices train in higher precision through the dequantized base.
Paged optimizers
Manage memory spikes during training.

Dettmers et al. 2023, Section 3.

Proving the adapter helps3 minThe release path and a paired comparison that is promising but not yet significant.

From adapter to release Diagram

Train adaptertune learning rate and data first
Beats best prompt?
- yes: to Regressions acceptable?
no
Stay with the promptor grow the set
Regressions acceptable?
- yes: to Merge or quantize
no
Fix data, then retrain
- retrain: back to Train adapter
Merge or quantizethe served form
Re-evaluatethe served artefact

"Beats" means a paired comparison on the same frozen items, never two separate pass rates. The last step scores the merged or quantized artefact that will actually be served.

Paired comparison on frozen set v1 (optional lab, synthetic) Chart

Best prompt passes5 items
Adapter passes8 items
Items gained4 items
Items regressed1 items

Four gains, one regression: an exact sign test on the five changed items gives p = 0.375. Promising, not proven; grow the set.

When LoRA is not enoughadvanced2 minSize of the change against the data you have.

When a small rank is enough (practitioner heuristic) Map

Small rank is usually fine

Plenty of data for a format change (claim summaries, report layout); extra rank rarely helps.

Test larger capacity

Large shift and lots of data (clinical notes): more matrices, higher rank or full tuning.

LoRA's sweet spot

Few examples, behaviour or format: small rank, careful data.

Data is the limit

Large shift with little data: neither method has much to learn from.

Consistent with Biderman 2024 (LoRA learns less on a large domain shift); confirm on a validation split.

Serving adapters2 minMerge, swap and test the artefact that ships.

Serving adapters Overview

Merge
Add BA into W: no inference overhead, one artefact per task.
Swap
Keep one base and load several adapters (Hugging Face PEFT docs, as read 2026-10-11).
Backend limits
Merging depends on the quantization backend (PEFT docs, as read 2026-10-11); check first.
Test what ships
Evaluate the merged or quantized form, not the training checkpoint.

Interview nuances and common misreadingsadvanced2 minFive claims about LoRA that sound right and what a staff-level answer says instead.

Common misreadings, and the staff-level correction Overview

0.1% of weights, 0.1% of risk
A small update applied in every layer can shift any behaviour; run the regression and safety suite.
Lower rank saves memory
The frozen base dominates; rank barely moves the total.
4-bit is lossless
It is an approximation; score the quantized artefact on your tasks.
Adapters stack freely
Two adapters trained separately can interfere when combined; evaluate the combination.
Training loss proves it
Only the frozen set against the best prompt decides.

Memory anchor

r x (d + k), not d x k: do the arithmetic, then evaluate the served artefact against the best prompt.

Useful for

Sizing an adaptation project before touching hardware, from parameters, adapter file size and approximate memory.
Choosing between full fine-tuning, LoRA and QLoRA from a memory budget and a frozen evaluation set (a hospital's on-premise GPU for discharge-note formatting is a typical 24 GiB case).
Serving several task variants from one base model by swapping adapters, such as one per client format at an insurer or one per product line at a retailer.

Not suitable for

Adding large amounts of new, changing knowledge; retrieval is the lever for that (Day 43, Day 48).
Predicting the quality of an adapter from its parameter count; quality is an empirical question per task and setting.

Traps

Reading memory estimates as capacity plans

A telecom team adapting a model to long network-fault logs reads the 12 GiB estimate for a 16-bit base as fitting on 24 GiB, but activations, long sequences and framework overhead push the real figure above the card, and the run fails with out-of-memory errors.

How to avoid it

Treat the arithmetic as a lower bound, measure peak memory on a short real run, and use gradient checkpointing, shorter sequences or a quantized base when it does not fit.

Evaluating the training checkpoint instead of the served artefact

A retailer's product-description adapter is scored unmerged in 16-bit, then merged and quantized for serving. The served model differs from the one that was evaluated, and quality or format compliance moves.

How to avoid it

Run the frozen evaluation set on the exact merged or quantized artefact and backend that production will use.

Senior reasoning

A senior does the arithmetic before booking hardware, then asks the evaluation question: what does the adapter do to the failures that motivated it (a bank's complaint-letter format, a claims summary that skips a field), and what does it break elsewhere? They compare against the best prompt and against retrieval, tune learning rate and data before rank, keep the adapter, base version and evaluation set version together as one release, and plan the retest when the base model changes.

Interview question

A bank's ML platform team knows full fine-tuning and asks whether LoRA should be the default for its document-assistant adapters. Explain LoRA to them and say when you would not use it.

Outline your answer first.

Recall cards scheduled after completion

How many trainable parameters does a rank-r LoRA update have for a d x k matrix, and what is the full count?
What is the difference between LoRA and QLoRA, and what must you evaluate?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, 2022core
Defines the frozen weight plus low-rank update BA, the initialisation (Gaussian A, zero B), the scaling by alpha over r and merging with no added inference latency; reports large reductions in trainable parameters and GPU memory on GPT-3 175B. (Section 4.1, Low-rank-parametrized update matrices)
QLoRA: Efficient Finetuning of Quantized LLMs — Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer, 2023core
Combines a frozen 4-bit NormalFloat base with LoRA adapters; double quantization reduces constants from 0.5 to about 0.127 bits per parameter, and paged optimizers manage memory spikes. (Section 3, QLoRA Finetuning)
Parameter-Efficient Transfer Learning for NLP — Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly, 2019optional
Bottleneck adapter modules as an earlier parameter-efficient method; they add computation at inference, which merged LoRA avoids.
LoRA (PEFT developer guide) — Hugging Facecore
Practical guide to LoRA configuration, merging an adapter into the base weights and loading several adapters on one base; merging support depends on the quantization backend.
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He, 2019optional
Source for the 16 bytes per parameter accounting of mixed-precision Adam training before activations. (Section 3.1)
LoRA Learns Less and Forgets Less — Dan Biderman et al. (12 authors), 2024optional
In code and maths settings LoRA underperformed full fine-tuning on the target domain but preserved base-model behaviour better; evidence that quality is task-dependent.
PEFT documentation — Hugging Faceoptional
PEFT library documentation and LoraConfig source; the defaults table (r 8, lora_alpha 8, dropout 0.0, use_rslora off) is as read 2026-10-12.
Training Deep Nets with Sublinear Memory Cost — Tianqi Chen, Bing Xu, Chiyuan Zhang, Carlos Guestrin, 2016optional
Gradient checkpointing, which trades recomputation for activation memory; named in the memory chapter.
Compressing LLMs: The Truth is Rarely Pure and Never Simple — Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, Yinfei Yang, 2024optional
Perplexity is an inadequate measure for compressed models; judge quantized models on task-specific evaluations.

Optional sections

Deep dive · 10 min

Why a low-rank update can be enough. Hu et al. hypothesise that the change in weights during adaptation has a low intrinsic rank, in the spirit of earlier findings that pretrained models can learn a task inside a small random subspace. It is a hypothesis that works in many settings, not a law: when the new task is far from pretraining the needed update may be higher rank, and quality at small $r$ falls behind full fine-tuning.

Reading the scaling $\alpha/r$ . The adapter's contribution is scaled by $\alpha/r$ . Keeping $\alpha$ fixed while varying $r$ keeps the update's magnitude comparable, which is why it is easy to sweep rank without retuning the learning rate completely. Learning rate and which matrices are adapted usually matter more than a larger rank (see insight-0548).

Memory accounting in full. For mixed-precision Adam: 2 bytes (16-bit weights) + 2 (gradients) + 4 (32-bit master copy) + 4 + 4 (first and second moments) = 16 bytes per trainable weight. With LoRA only adapter weights carry the gradient and optimizer terms, so the 16-byte factor applies to millions of weights rather than billions, while the frozen base still costs its storage (2 bytes in 16-bit, about 0.52 bytes in 4-bit with double quantization). Activations scale with batch size and sequence length and are reduced by gradient checkpointing at the price of recomputation.

Serving adapters. Merging $BA$ into $W$ gives a single standard model with no extra latency. Keeping adapters unmerged allows one base to serve many tasks but adds a small computation per adapted matrix. Merging into a quantized base can be restricted by the backend (see the PEFT guide), so decide the serving form before evaluating.

Retrieval project link (weekend studio). Sunday's recall block asks for the parameter count of a rank-8 LoRA update on a 4096 x 4096 matrix compared with the full matrix. The answer from today's formula is 8 x (4096 + 4096) = 65,536 against 16,777,216. The adaptation decision record there should cite that arithmetic and state which evidence would make fine-tuning preferable to retrieval for the study assistant.

Hands-on lab: Rank, placement and a paired comparison against the Day 43 baseline · 15 min

Run the code. Part 1 shows that placing adapters on more matrices changes the trainable count as much as raising the rank. Part 2 compares a hypothetical adapter with the Day 43 baseline on the same 12 items using an exact sign test on the changed items. Edit the adapter's results so that it regresses on two items and gains on three: how does the p-value move? Then write one sentence for the release note that is honest about what 12 items can and cannot show.

from math import comb

# Part 1: which matrices you adapt matters as much as the rank (hypothetical 32-layer, width-4096 model)
d, layers = 4096, 32
shapes = {"q_proj": (d, d), "k_proj": (d, d), "v_proj": (d, d), "o_proj": (d, d), "mlp_up": (d, 4 * d), "mlp_down": (4 * d, d)}
base = 12 * layers * d * d
for name, targets in {"q,v": ["q_proj", "v_proj"], "all attention": ["q_proj", "k_proj", "v_proj", "o_proj"], "attention + mlp": list(shapes)}.items():
    for r in (8, 32):
        n = sum(r * (a + b) for a, b in (shapes[t] for t in targets)) * layers
        print(f"targets={name:<16} r={r:<2} trainable={n:>11,} ({n / base:.2%} of base)")

# Part 2: compare an adapter with the Day 43 baseline on the SAME 12 items (hypothetical recorded results)
ids = [f"e{i:02d}" for i in range(1, 13)]
baseline = dict(zip(ids, [0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1]))  # best prompt: 5/12 (Day 43)
adapter = dict(zip(ids, [0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0]))  # hypothetical LoRA run
wins = [i for i in ids if adapter[i] > baseline[i]]
losses = [i for i in ids if adapter[i] < baseline[i]]
n = len(wins) + len(losses)
tail = sum(comb(n, j) for j in range(0, min(len(wins), len(losses)) + 1)) / 2 ** n
print(f"baseline {sum(baseline.values())}/12, adapter {sum(adapter.values())}/12; gains {wins}; regressions {losses}")
print(f"exact two-sided sign test on {n} changed items: p = {min(1.0, 2 * tail):.3f}")
print("still failing after adaptation:", [i for i in ids if not adapter[i]])

Interview practice · 10 min

Your manager at a logistics company says "LoRA only trains 0.1% of the weights, so it is basically free and safe" about an adapter for the shipment-exception assistant. Respond.

Answer as you would in the interview.

Content version 6 · last verified 2026-10-18 · depends on numpy