Introduced on day 46
Autoregressive model
A model that factorises the data into an ordered sequence and predicts each element from the earlier ones, trained by maximum likelihood.
In detail
The joint probability is written as a product of next-element conditionals, so the likelihood of a sequence is exact and sampling proceeds one element at a time. Large language models are autoregressive over tokens; the same idea applies to pixels, audio samples and table columns. Errors can compound along a long sequence, and sampling cost grows with output length.