Skip to content
ML Academy

Introduced on day 38

Anisotropy (in embeddings)

Embedding vectors occupy a narrow cone rather than spreading over the whole space, so even unrelated pairs have high cosine similarity.

In detail

It makes raw cosine values hard to interpret and thresholds hard to transfer between models. Centering the vectors or whitening them are common mitigations, and the check is the similarity distribution of random or labelled negative pairs.