Introduced on day 42
Pretraining
The first, large-scale training stage in which a language model learns to predict the next token over a very large text collection, before any task-specific adaptation.
In detail
Pretraining fixes most of what a model can do and know up to its data cutoff. Later stages (supervised fine-tuning, preference tuning) adjust behaviour on top of those weights; they are cheap compared with pretraining, and they cannot reliably add large amounts of new, changing knowledge.