Neural networks, deep learning, NLP · insight-1444
Centre or whiten sentence embeddings before nearest-neighbour search and compare with the raw vectors
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data scientist, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Scores between unrelated texts from a BERT-based encoder are all high, and every query seems to retrieve similar neighbours.
Action
Subtract the training-set mean, optionally remove top components or whiten, apply the identical transform to queries, and compare retrieval on labelled pairs with and without it.
Reason
Ethayarajh found contextualised representations are not isotropic in any layer; Su et al. report that whitening improves isotropy and is competitive, and can reduce dimensionality.
Boundary
Encoders not trained for similarity, and small domain corpora.
Symptom or risk
A shared mean direction and a few dominant components dominate cosine, compressing the usable score range, so ranking and thresholds respond mostly to the common component.
Exceptions
Contrastively trained sentence encoders are often less affected; the labelled pairs decide.
Trade-offs
Whitening amplifies noise in low-variance directions and needs enough data to estimate.
Pitfalls
Fitting the transform on the evaluation queries; changing the transform without re-indexing.
Lessons and concepts
Claims and their sources
Ethayarajh reports that contextualized word representations of all words are not isotropic in any layer of BERT, ELMo and GPT-2.
Su et al. (arXiv preprint) report that whitening improves the isotropy of BERT sentence representations, performs competitively, and can reduce dimensionality for faster retrieval.