LLMs, GenAI, RAG, agents, evaluation · insight-1631
Do not extrapolate small-model 8-bit results to large models, because outlier features break naive quantisation
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A model that quantised cleanly at a few billion parameters is scheduled for the same naive 8-bit recipe at a much larger size.
Action
Evaluate the specific quantisation method on the specific model and size using the task eval set, and prefer methods that handle outlier channels, such as mixed-precision decomposition, over plain per-tensor rounding.
Reason
Dettmers et al. report systematic outlier features in large transformers that dominate predictions; handling them in a 16-bit path while multiplying over 99.9% of values in 8-bit preserved quality in their tests.
Boundary
Self-hosted transformer inference and QLoRA-style loading where memory forces lower precision.
Symptom or risk
Accuracy collapses on the larger model, and the failure is blamed on the data or the fine-tune instead of the quantisation method.
Exceptions
Hosted APIs, where the provider chooses the serving precision.
Trade-offs
Outlier-aware kernels add complexity and may run slower than fully uniform ones on some hardware.
Pitfalls
Reporting only perplexity or only the average score, which can hide a collapse on knowledge-heavy slices.
Lessons and concepts
Claims and their sources
Dettmers et al. report that large transformers have systematic emergent outlier features and that isolating them in a 16-bit matrix multiplication lets more than 99.9% of values be multiplied in 8-bit without performance degradation on their 175B-scale tests.