Introduced on day 44
Quantization
Storing or computing with weights in fewer bits (for example 8-bit or 4-bit instead of 16-bit) to save memory and bandwidth, at the cost of approximation error.
In detail
Weights are grouped into blocks that share scaling constants, and values are rounded to a small set of levels. Effects on quality vary by model, bit width and task, and aggregate scores such as perplexity can hide losses on knowledge-intensive tasks, so evaluate the quantized artefact you will actually serve.