S32 · paper simulation

LoRA matches full fine-tuning: how much can 400 invoices show?

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
35 minabout
0stages committed

The brief

Your team plans to fine-tune a language model for invoice-field extraction and must pick between full fine-tuning and a LoRA adapter. A colleague forwards a preprint. The paper and its results are fictional, written for this exercise. Its abstract (paraphrased):

"On InvoiceFields-400, a rank-4 LoRA adapter matches full fine-tuning (89.5% against 90.5% exact-match accuracy) while training 0.8% of the parameters, showing that parameter-efficient tuning loses nothing on extraction."

Facts from the methods section (fictional): the adapter's rank and learning rate were chosen from 12 configurations using test accuracy; full fine-tuning was run once with a default learning rate; one random seed each; the test set's invoices come from the same vendors as the training set; the item-level outcomes are given (below).

	Full fine-tuning correct	Full fine-tuning wrong
LoRA correct	346	12
LoRA wrong	16	26

Stage 1 of 4

The paired outcomes give LoRA 358 of 400 (89.5%) and full fine-tuning 362 of 400 (90.5%). The 28 items where the two disagree decide whether they differ: LoRA wins 12, full fine-tuning wins 16.

What can the 400-item comparison support?

The paired difference is 1.0 point for full fine-tuning, interval about -1.6 to +3.6 points (McNemar exact p = 0.57); equivalence within 2 points or no loss cannot be shown.The two systems are equivalent: the difference is 1 point and p = 0.57, so the null hypothesis of no difference is retained, and the authors' reading that LoRA loses nothing holds.Full fine-tuning is better: 362 correct against 358 is a gain, and a one-point gap is large enough to matter at the volumes of invoice processing.Nothing can be said, because 400 items from a single dataset in a single run is too little for any comparison of fine-tuning methods.