Neural networks, deep learning, NLP · insight-0455
Compare tokenizer fertility before choosing a multilingual encoder for a non-English task
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, Applied AI, ML engineer
- Industries
- aviation, telecom
Summary: situation, action, reason, boundary
Situation
You must classify Turkish customer complaints and are choosing between a multilingual encoder and a model pretrained mainly on Turkish text.
Action
On a sample of your own text, compute subwords per word and the share of words split into three or more pieces for each candidate tokenizer. Shortlist on that, then compare the candidates on the same split and metric.
Reason
Rust et al. found a dedicated monolingual tokenizer matters about as much as pretraining data size for downstream performance, and that well-represented languages lose little in multilingual models.
Boundary
Classification, tagging and extraction in languages other than English, or in mixed-language data.
Symptom or risk
If the multilingual vocabulary covers the language poorly, words shatter into many subword pieces; accuracy suffers and more examples hit the maximum length, and the model is blamed instead of its tokenizer.
Exceptions
High-resource languages that are well covered by the multilingual vocabulary; cross-lingual transfer needs may still favour a multilingual model.
Trade-offs
A monolingual model may handle code-switching and foreign names worse.
Pitfalls
Measuring fertility on clean news text when your inputs are informal, misspelled messages.
Lessons and concepts
Claims and their sources
Rust et al. find a designated monolingual tokenizer plays a role in downstream performance comparable to pretraining data size, and that languages adequately represented in a multilingual vocabulary show negligible performance decreases.
Petrov et al. report tokenization-length disparities between languages persist even for tokenizers intentionally trained for multilingual support.