Neural networks, deep learning, NLP · insight-1449
Do not use raw BERT [CLS] or mean-pooled outputs as sentence embeddings for similarity
- Evidence type
- empirical finding
- Difficulty
- easy
- Roles
- Data scientist, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A prototype embeds sentences with a base BERT checkpoint, takes the [CLS] vector or the mean of the last layer, and ranks by cosine.
Action
Use an encoder trained for sentence similarity, or fit a task head on labelled pairs; if raw BERT features feed a supervised classifier, judge them by that classifier on held-out data, not by cosine.
Reason
The Sentence-BERT authors argue that averaging BERT outputs or using the [CLS] token yields poor sentence embeddings, often worse than averaged GloVe vectors, which motivated training siamese networks.
Boundary
Unsupervised similarity, clustering and retrieval with untuned encoders.
Symptom or risk
The ranking is poor, the team blames the idea of embeddings, and a better similarity model is never tried.
Exceptions
Supervised probes on frozen BERT features can work well, since the classifier learns the geometry.
Trade-offs
Sentence encoders have length limits and their own domain gaps.
Pitfalls
Reading probe accuracy as proof that cosine geometry is good.
Lessons and concepts
Claims and their sources
Reimers and Gurevych report that averaging BERT output vectors or using the [CLS] token gives rather poor sentence embeddings, often worse than averaging GloVe vectors.