Neural networks, deep learning, NLP · insight-1446
Measure pretrained word-vector coverage on your own tokens, by slice, before relying on static embeddings
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data scientist, Applied AI
- Industries
- aviation, telecom
Summary: situation, action, reason, boundary
Situation
GloVe or word2vec vectors feed a complaint classifier, and the text is full of flight codes, brand names and inflected Turkish forms.
Action
Compute the share of tokens and of documents with a vector, by language and channel, list the most frequent missing tokens, and fix with subword vectors or character n-grams, with a TF-IDF fallback below a coverage floor.
Reason
Whole-word vectors cannot cover unseen words, whereas Bojanowski et al. build vectors from character n-grams so that rare and unseen words still get a representation.
Boundary
Static word-vector pipelines on domain or morphologically rich text.
Symptom or risk
Unknown tokens are dropped or mapped to zeros, so many documents are averaged from a few known words and the slice with the lowest coverage quietly gets the weakest features.
Exceptions
Contextual encoders with subword vocabularies have no hard out-of-vocabulary tokens, though pieces may be poor.
Trade-offs
Subword vectors are larger and slower to load.
Pitfalls
Reporting only global coverage; counting tokens, not documents, when most documents have a gap.
Example
Hypothetical: 92 percent of English tokens but 61 percent of Turkish tokens have a vector, so the Turkish slice averages far fewer words.
Lessons and concepts
Claims and their sources
Bojanowski et al. represent words as bags of character n-grams so that rare and unseen words obtain vectors, which is stated to matter especially for morphologically rich languages.