Introduced on day 44
QLoRA
LoRA training on top of a frozen base model stored in 4-bit precision, which cuts the memory needed to fine-tune large models while the adapters are trained in 16-bit.
In detail
It combines a 4-bit NormalFloat data type, double quantization of the quantization constants and paged optimizers. The saving comes from the frozen base weights; the adapter, its optimizer state and the activations still use memory. Because the base is approximated, the served artefact should be evaluated on your own tasks rather than assumed equal to the 16-bit model.