Neural networks, deep learning, NLP · insight-1448
Remove the common component before averaging static word vectors into text features
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Short texts are represented by the plain mean of GloVe or word2vec vectors for a classifier or nearest-neighbour search.
Action
Subtract the mean word vector and project out a few top principal directions estimated on the training vocabulary before averaging, and compare with the plain mean and with TF-IDF.
Reason
Mu et al. report in an arXiv preprint that removing the shared mean and a few dominant directions improved word-similarity and sentence-level tasks across datasets, representation methods and languages.
Boundary
Static word vectors on small labelled sets, as in the TF-IDF versus embeddings project.
Symptom or risk
Frequent words and a shared mean direction dominate the average, texts all look alike, and rare content words barely register.
Exceptions
Contextual and sentence encoders already differ in geometry; test, do not port.
Trade-offs
Number of removed directions is a tuned setting; extra code at serving.
Pitfalls
Estimating directions on test data; removing so many that content is lost.
Lessons and concepts
Claims and their sources
Mu, Bhat and Viswanath (arXiv preprint, not peer reviewed) report that removing the mean vector and top directions from word vectors improves word similarity and sentence-level tasks including semantic textual similarity and text classification.