Introduced on day 41
Tokenization
Splitting raw text into the units a model counts or embeds; word-level for classical NLP, subword units (byte-pair encoding and similar) for language models.
In detail
The tokenizer decides what the columns of a bag-of-words matrix are, or which integer ids a language model sees. Lowercasing, punctuation handling and n-grams change a classical model's features; a subword vocabulary changes sequence length and therefore cost and context use (Day 42). A tokenizer is fixed at training time, so text processed with a different tokenizer at serving time silently degrades results.