Neural networks, deep learning, NLP · insight-0468
Use contextual embeddings when word sense changes the label
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, Applied AI
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
You classify short airline messages with averaged static word vectors, and words like "hold" (baggage hold, on hold, hold a seat) carry the meaning.
Action
Compare averaged static vectors with a TF-IDF baseline and a contextual sentence-embedding model on the same split, and inspect errors on messages containing known ambiguous terms.
Reason
Static methods such as word2vec learn one vector per word, while contextual models produce different representations of the same word in different contexts.
Boundary
Short-text classification and search where polysemy matters.
Symptom or risk
A static vector gives each word one representation for all senses, so messages with the same words but different meanings look alike and errors cluster on ambiguous terms.
Exceptions
Long documents with consistent vocabulary, where static or lexical features work well and run cheaply.
Trade-offs
Contextual models cost more compute and latency per message.
Pitfalls
Attributing errors to small data when they concentrate on ambiguous words.
Lessons and concepts
Claims and their sources
Mikolov et al. learn one continuous vector per word with CBOW and skip-gram.
Ethayarajh reports that, on average, less than 5% of the variance in a word's contextualized representations can be explained by a static embedding.