Introduced on day 42
Token
The unit of text a language model reads and writes: a word, a piece of a word or a short run of digits drawn from a fixed vocabulary, mapped to an integer id.
In detail
Subword tokenizers split rare words into several tokens, so token counts differ from word counts and differ by language. Context limits, latency and per-token pricing are all measured in tokens, and a token is not necessarily a meaningful unit: an identifier such as a flight number may be split into several arbitrary pieces.