Neural networks, deep learning, NLP · insight-0463

Do not reuse a cosine-similarity threshold across embedding models

Situation
A duplicate-ticket detector flags pairs above cosine 0.8, and the team swaps in a new embedding model without changing the threshold.
Action
For each model, plot similarity distributions for labelled duplicate and non-duplicate pairs, and set the threshold from the target precision or recall on that labelled set.
Reason
Contextual word representations have been found to be anisotropic, and cosine values can be arbitrary for some regularized models, so raw similarity numbers are not calibrated across embedding models; the thresholds must be re-estimated rather than carried over.
Boundary
Deduplication, near-match alerts, retrieval cut-offs and clustering thresholds.

Symptom or risk

Similarity distributions differ between models, often with most pairs scoring high, so the old threshold flags far too many or too few pairs.

Rank-based uses, such as top-k retrieval, are less sensitive to absolute scale.

Labelling pairs costs effort; a few hundred pairs usually suffice to place a threshold.

Interpreting 0.8 as 80% similar.

Ethayarajh found contextualized word representations are not isotropic in any layer of ELMo, BERT and GPT-2.
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
Steck et al. derive that for some regularized linear models cosine similarities can be arbitrary.
Is Cosine-Similarity of Embeddings Really About Similarity?