Core day 38 of 60 · Week 8 · Part II · application depth

Embeddings and Representation Learning

An airline wants support search to match "my suitcase never showed up" to the article "Delayed baggage claim". A vendor demo reports "cosine similarity 0.82" for that pair. Before it goes into production: what does 0.82 mean, what do you compare it against, and what could break on your airline's own vocabulary?

By the end: You compute a cosine similarity by hand, read a small retrieval result against a lexical baseline using labelled queries, and name the domain, evaluation and geometry caveats before trusting a similarity score.

At a glance

An embedding places items so that closeness reflects shared context in the training data; a similarity score means something only against a baseline on labelled queries.

Dense vectors
Tens to a thousand numbers per item instead of one sparse axis per word.
Cosine similarity
Dot product over both lengths, in [-1, 1]; ignores vector length.
Context not definition
Neighbours reflect what co-occurs in the training corpus.
Evaluate on your task
Labelled queries, BM25 baseline, precision at k, slices by query type.
Three caveats
Domain shift, anisotropy (a narrow cone) and hubness (crowding neighbours).

Recall · 5 min

No notes: (1) the dot product of $(1, 2, 2)$ and $(2, 2, 1)$ and the length of $(1, 2, 2)$ ; (2) why one-hot encoding a 300-level airport code makes every pair of airports equally different (Day 9); (3) what PCA does to a cloud of points when it keeps two components (Day 14).

Open due reviews

Concept · 10 min

In one-hot or TF-IDF space "suitcase" and "luggage" share no coordinate, so their similarity is zero. An Embedding is a short dense vector, learned so that items used in similar contexts land close. The map analogy breaks quickly: an embedding has hundreds of dimensions, and closeness means only the relation the training objective rewarded.

Neighbours reflect the text the vectors were built from - closeness is co-occurrence in that corpus, not a definition

Predict first

In these academy-text vectors, which term is the nearest neighbour of "capacity"?

Text description of the figure

A 2-D map of term vectors computed from this academy's own text with latent semantic analysis (not a neural embedding model), with a chosen query term and its nearest neighbours by cosine in the full space. The map is a precomputed projection.

Sparse word axes versus a dense embedding Comparison

Criterion	One-hot or TF-IDF	Dense embedding
Size per item	One axis per word or level: thousands, mostly zeros	Typically 50 to 1,000 numbers
suitcase vs luggage	0.00 if no document contains both (today's run)	0.99 in today's 3-dimensional LSA space
Learned from	Counts only; nothing is learned	Co-occurrence, prediction or contrastive pairs
Typical failure	Misses synonyms and paraphrases	Misses jargon and codes absent from its training text

Cosine similarity, with a worked example Formula

\cos(u, v) = \frac{u \cdot v}{\lVert u \rVert\, \lVert v \rVert}, \qquad u = (3, 4),\; v = (4, 3): \; \frac{24}{5 \cdot 5} = 0.96

$u \cdot v$: dot product: 3 x 4 + 4 x 3 = 24
$\lVert u \rVert$: length: square root of 9 + 16 = 5
$[-1, 1]$: 1 same direction, 0 orthogonal, -1 opposite

For unit-length vectors cosine equals the dot product. Some models are tuned for the raw dot product; use the score the model was trained with.

Sparse cosine, LSA cosine and a 2-D picture of the application's words Illustration

Sparse space: two words are similar only if they occur in the same snippets. "suitcase" and "luggage" never do: cosine 0.00.

Three latent dimensions (52% of the squared norm): words used next to the same words ("baggage", "carousel") now point the same way, so each topic becomes one block.

baggage
refunds
check-in

A 2-D PCA picture of the 3-d word vectors (keeps 99% of their variance). Cosines on the lines are from the full 3-d space, not from the picture.

The nine synthetic FAQ snippets of the application block; TF-IDF and truncated SVD recomputed in TypeScript with the default settings of scikit-learn (exact SVD, so signs may differ from the randomized solver; cosines do not).

Three latent dimensions turn "never in the same snippet" into "used next to the same words". The map is drawn from the vectors; the cosines come from the vectors, not the map.

Where the vectors come from Stack

Count-based (LSA)Truncated SVD of a term-document matrix: today's application code.
Static word vectorsword2vec predicts a word from its neighbours; GloVe fits global co-occurrence counts.
Contrastive encodersPull an anchor toward its positive and away from negatives; whole sentences.
Task-trained lookup tablesOne learned row per category level (merchant, SKU, postcode), trained with the task loss.

Evaluating embeddings on your own task Diagram

Labelled queriesreal chats, known answers
- to Dense model
Lexical baselineBM25 or TF-IDF
- to Precision or recall at k
Dense modelsame queries, same corpus
Precision or recall at kper model
Slicescodes, jargon, long queries
Gap larger than noise?
- no: to Try a hybrid
yes
Adopt the winnerthreshold from labelled pairs
Try a hybridfuse both ranked lists

0.82 on a demo pair says nothing until the model is compared with BM25 on your own labelled queries, by slice. Any threshold is set per model, from the scores of labelled positive and negative pairs.

Three caveats before trusting a similarity score Overview

Domain shift
A web-trained encoder may not know your SKUs, policy codes, ICD codes or jargon.
Anisotropy
Vectors in a narrow cone: unrelated pairs still score high; thresholds do not transfer.
Hubness
A few points sit in many neighbour lists and crowd out the rest.
2-D pictures
A t-SNE or PCA map is a lossy projection, not evidence of quality.

Sources of vectors: count-based (LSA, truncated SVD of a term-document matrix), predictive (word2vec), co-occurrence-fitted (GloVe) and contrastive sentence encoders.
The standard score is Cosine similarity, which ignores length; use the score the model was trained for.
For categorical features, a lookup table with one $d$ -vector per level replaces one-hot encoding.
Judge embeddings on your task: labelled queries, a lexical baseline (BM25 or TF-IDF), precision or recall at $k$ , slices.
Caveats to name: domain shift, Anisotropy (in embeddings) and Hubness. A 2-D projection is a picture, not evidence.

Assumptions

The relation the training objective rewarded (topic, paraphrase, entailment) is the relation your task needs.
Queries and documents resemble the data the encoder was built from; codes, numbers and jargon may be split or ignored.
A labelled evaluation set exists. Without one a similarity score is an anecdote.

Application and workplace decision · 10 min

A search lead wants to know whether a dense representation beats TF-IDF before buying a vendor model.

Data: nine synthetic FAQ snippets in three topics (baggage, refunds, check-in), nine one-word queries with a known topic.
The code compares TF-IDF with TF-IDF plus truncated SVD (LSA) at several dimensions and scores precision at 3.
It prints the cosine between two word pairs in both spaces.

Before running, predict:

(a) whether "suitcase" and "luggage" are similar in raw TF-IDF space, given that no snippet contains both;
(b) whether more latent dimensions always help.

Then state what nine queries can and cannot establish.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
data = """bag: my suitcase did not arrive at the baggage carousel
bag: luggage delayed so file a baggage claim at the carousel desk
bag: lost luggage claim form and baggage tracing number
refund: refund for a cancelled flight returned to the card
refund: request a refund or a travel voucher after cancellation
refund: voucher or money back for a cancelled booking
checkin: online check-in opens and the boarding pass is on the app
checkin: print the boarding pass at a kiosk or check in online
checkin: mobile boarding pass and seat selection at the kiosk"""   # synthetic FAQ snippets
topics, texts = map(np.array, zip(*(line.split(": ") for line in data.split("\n"))))
queries = {"suitcase": "bag", "luggage": "bag", "tracing": "bag", "voucher": "refund", "cancelled": "refund",
           "money": "refund", "kiosk": "checkin", "boarding": "checkin", "mobile": "checkin"}
vec = TfidfVectorizer().fit(texts)
D, Q = vec.transform(texts), vec.transform(list(queries))
def precision_at_3(sim):
    top = np.argsort(-sim, axis=1, kind="stable")[:, :3]
    return np.mean([(topics[t] == g).mean() for t, g in zip(top, queries.values())])
print("TF-IDF precision@3:", round(precision_at_3(cosine_similarity(Q, D)), 2))
for k in (2, 3, 6):
    svd = TruncatedSVD(n_components=k, random_state=0).fit(D)
    sim = cosine_similarity(svd.transform(Q), svd.transform(D))
    print(f"LSA k={k} precision@3:", round(precision_at_3(sim), 2))
svd = TruncatedSVD(n_components=3, random_state=0).fit(D)
idx = {t: i for i, t in enumerate(vec.get_feature_names_out())}
sparse, dense = D.T.toarray(), svd.components_.T          # one row per word
for a, b in [("suitcase", "luggage"), ("suitcase", "voucher")]:
    cs = cosine_similarity(sparse[[idx[a]]], sparse[[idx[b]]])[0, 0]
    cd = cosine_similarity(dense[[idx[a]]], dense[[idx[b]]])[0, 0]
    print(f"{a} vs {b}: TF-IDF {cs:.2f}, LSA {cd:.2f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 33 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The object, defined3 minAn embedding as a learned matrix and a similarity function, the lookup computed by hand, and the assumptions written as properties of the loss.

An embedding, defined: a learned lookup, judged by a similarity function Formula

e_k = E^{\top} \mathbf{1}_k = E[k,:] \in \mathbb{R}^{d}, \qquad E \in \mathbb{R}^{K \times d}, \qquad \text{sim}(k, k') = \cos(e_k, e_{k'})

$\mathbf{1}_k$: one-hot vector of item k (a word, a SKU, a merchant): K long, a single 1
$E$: the embedding matrix, K rows of d numbers; the only thing learned
$E^{\top} \mathbf{1}_k$: multiplying by a one-hot just selects row k, so a lookup is a linear layer with no bias
$d$: embedding size, far smaller than K (16 to 1,024 against thousands to millions of items)
$\text{sim}$: the score used to compare items; it must match the score the training objective optimised

For text encoders e is a network instead of a table, but the object is the same: a map from items to R^d whose geometry is whatever minimised the training loss. The loss, not the word "embedding", decides what close means.

A lookup is a matrix product, by hand (K = 4 items, d = 2) Sequence

The table
E has rows (0.2, -0.1), (0.5, 0.3), (-0.4, 0.8), (0.0, 0.6): one 2-vector per item
One-hot input
Item 3 is (0, 0, 1, 0)
The product
(0, 0, 1, 0) E = 0 x row 1 + 0 x row 2 + 1 x row 3 + 0 x row 4 = (-0.4, 0.8)
The gradient
Only row 3 multiplied a non-zero input, so only row 3 gets a gradient from this example
The consequence
A level seen 5 times gets 5 updates; a rare level keeps a vector close to its random start

Verified in numpy. Libraries skip the multiplication and index the row directly; the gradient pattern is the same.

What the matrix and the loss commit you to Overview

The objective is the relation
Trained on co-occurrence, close means co-occurring, not equivalent: a substitute and a complement both land close.
Distribution match
An item never seen in training has no trained row, or is split into pieces trained elsewhere.
A meaningful score
Cosine, dot product and Euclidean distance rank items differently unless vectors are unit length.
Cost
A lookup is O(d) per item; brute-force retrieval is O(N d) per query, hence vector indexes (Day 48).

Cosine from dot product and norms, by hand3 minThe definition derived from the angle between vectors, one example worked on paper, and how cosine, dot product and distance relate.

Cosine by hand on two product vectors (illustrative 4-d e-commerce embeddings) Sequence

Vectors
u = (2, 1, 0, 2) for a trail shoe, v = (1, 2, 1, 1) for a road shoe
Dot product
u . v = 2 x 1 + 1 x 2 + 0 x 1 + 2 x 1 = 6
Norms
|u| = sqrt(4 + 1 + 0 + 4) = 3; |v| = sqrt(1 + 4 + 1 + 1) = sqrt(7) = 2.646
Cosine
6 / (3 x 2.646) = 6 / 7.937 = 0.756
Length does not count
Triple v: the dot product becomes 18, |3v| = 7.937, the cosine stays 0.756
Unit vectors
Normalise both: dot = cosine, and squared distance = 2 - 2 cos = 0.488, so distance 0.699
So what
On unit vectors, ranking by cosine, by dot product and by Euclidean distance gives the same order

Verified in numpy. If a model was trained with the raw dot product, vector length carries signal (popularity, confidence) and normalising throws it away.

Where cosine comes from, and why indexes normalise once Formula

\lVert \hat u - \hat v \rVert^2 = 1 + 1 - 2\cos\theta

$\cos\theta$: from u . v = |u| |v| cos(angle): dividing the dot product by both lengths leaves only the angle
$\hat u, \hat v$: unit-length versions: their dot product is the cosine and their squared distance is 2 - 2 cos
$[-1, 1]$: the general range; vectors with only non-negative entries (TF-IDF, counts) score 0 to 1, so sparse cosines are never negative

Because distance and cosine are tied on unit vectors, many vector indexes normalise once at insert time and then rank by the cheaper dot product.

How contrastive encoders learn similarityadvanced5 minThe InfoNCE loss term by term, one example at three temperatures by hand, its gradient, and why the choice of positives defines similarity.

The contrastive loss behind many sentence encoders Formula

L = -\log \frac{e^{s(a,p)/\tau}}{e^{s(a,p)/\tau} + \sum_j e^{s(a,n_j)/\tau}}

$a, p$: anchor and its positive (a paraphrase, a clicked answer)
$n_j$: negatives, often the other items in the batch
$s$: similarity, usually cosine
$\tau$: temperature: smaller values sharpen the softmax over candidates

It is a softmax cross-entropy (Day 32) over one positive and many negatives. What counts as a positive defines what "similar" will mean.

InfoNCE by hand: positive 0.8, hard negative 0.7, easy negative 0.1 Table

Temperature	Softmax p (pos, hard, easy)	Loss -ln p_pos	Gradient dL/ds = (p - y) / tau	Hard negative's share of the push
tau = 1.0	0.416, 0.377, 0.207	0.876	-0.58, +0.38, +0.21	65%
tau = 0.5	0.484, 0.396, 0.119	0.725	-1.03, +0.79, +0.24	77%
tau = 0.1	0.731, 0.269, 0.001	0.314	-2.69, +2.69, +0.01	99.8%

Each row: divide the similarities by tau, exponentiate, normalise (Day 32 softmax), then take minus the log of the positive's share. The gradient is the softmax cross-entropy p - y, scaled by 1 / tau. Computed in numpy; the temperature demo shows the same effect over many negatives.

Temperature decides which negatives the loss cares about Illustration

Loss 2.34. The hard negative takes 10% of the push on the 15 negatives; the easy ones share the rest almost evenly.

Loss 1.07. The hard negative takes 37% of the push on the 15 negatives.

Loss 0.22. The hard negative takes 99% of the push on the 15 negatives: training is almost only about separating it from the positive.

Seeded similarities: positive 0.62, one hard negative 0.55, 14 easy negatives around 0.1. Each panel is the softmax of similarity / tau over the same 16 candidates; loss = -log of the positive's weight.

High temperature spreads the push over all negatives; low temperature concentrates it on the hardest one. A wrongly labelled "negative" that is really a paraphrase then dominates training.

One contrastive training step, as a loop Loop

Pairs (a, p)clicks or paraphrasesencode ↓
Scores s / tauvs positive and batchsoftmax ↓
Softmax over 1 + Nclass = the positive-ln p_pos ↓
Gradient (p - y) / taupush = softmax shareupdate ↓
Encoder updatepull p in, push n outnext batch ↓
↺ back to 1 · Pairs (a, p)

The loss is low when the positive wins the softmax, whatever its absolute similarity. As tau falls, the hardest negatives take almost all of the push (the table above shows 65% to 99.8%).

Three things the contrastive setup decides Overview

Positives are the definition
Clicked answers teach 'what users click'; paraphrase pairs teach 'same meaning'. The two disagree.
In-batch negatives are noisy
A batch can hold a true paraphrase labelled negative; a low tau makes it the loudest signal.
More negatives
Minimising InfoNCE maximises a lower bound on mutual information between the views (Oord et al., 2018).

Distance concentration and what it does to neighboursadvanced4 minWhy random high-dimensional distances bunch together, derived from a sum of independent terms, checked on seeded vectors, and tied to hubness and thresholds.

Distance concentration: why high-dimensional neighbours are all nearly equally far Formula

\frac{\mathrm{SD}\,\lVert x - q \rVert^2}{\mathbb{E}\,\lVert x - q \rVert^2} = \frac{\sqrt{8d}}{2d} = \sqrt{2/d}

$x, q$: a point and a query, each with d independent N(0, 1) coordinates
$\lVert x - q \rVert^2$: a sum of d terms (x_c - q_c)^2, each with mean 2 and variance 8
$\sqrt{2/d}$: relative spread of squared distances: 0.14 at d = 100, 0.05 at d = 768
$\cos(u, v)$: u . v has mean 0 and variance d; each norm is close to sqrt(d), so the cosine has SD close to 1 / sqrt(d)

Sums of many independent terms concentrate around their mean (the law of large numbers at work). Real embeddings are not independent Gaussians, but the direction of the effect holds: raw distances separate neighbours less as d grows, which feeds hubness.

Nearest and farthest converge: distances and cosines as dimension grows Illustration

d = 2
d = 16
d = 128
d = 768

Relative contrast (farthest - nearest) / nearest: d = 2: 57.42, d = 16: 1.63, d = 128: 0.40, d = 768: 0.13. In high dimensions the nearest and the farthest point are almost equally far from the query.

Cosine of two independent random vectors has mean 0 and standard deviation close to 1 / sqrt(d): measured 0.037 at d = 768 against 0.036 predicted. "Unrelated" means a cosine near 0 only for isotropic vectors.

Seeded synthetic Gaussian vectors, not a trained embedding model. Real embeddings are not isotropic (see the anisotropy figure), but the concentration of distances in many dimensions is a general effect.

The left panel divides every distance by the median, so a spike at 1 means every point is about equally far. The right panel checks the 1 / sqrt(d) spread of random cosines derived above.

What distance concentration does downstream Overview

Offsets decide rankings
All candidates almost equally far: a shared direction or a popular item decides the order. Hubs grow here.
Thresholds lose meaning
768-d isotropic: random cosines have SD 0.036, so 0.3 is far above chance; in an anisotropic model it can be average.
The worst case
Trained embeddings sit near a much lower-dimensional structure, which is why they work at all.

The derivation above is the worst case (independent coordinates), not a forecast for a trained model; the direction of the effect still holds.

Embeddings for categorical features2 minA learned table instead of one-hot columns, its parameter count, and what to do with rare levels.

Embedding tables for categorical features Formula

N = K \cdot d = 1{,}000 \times 16 = 16{,}000

$K$: number of levels of the category (1,000 merchants here)
$d$: embedding size, chosen by you

A lookup replaces a 1,000-column one-hot block. The rows are learned with the task loss, so similar merchants end up close only if that helps the task (for example, fraud scoring).

A lookup table learns only the rows it sees Hierarchy

A category levelone learned row of d numbers
- many rowsFrequent levelmany updates: a learned vector
- few rowsRare levelstays near its random start
  - fixGroup into one bucket
  - fixHash into fewer rows
  - fixStart from target encodingDay 9
- padding_idxPadding or unknownno gradient in PyTorch: keeps its vector

Only rows that multiply a non-zero input get a gradient (the lookup steps above), so data per level, not the table size, decides what is learned.

Reading today's result2 minWhat nine labelled queries show and what they cannot.

Today's precision at 3 on nine labelled queries Chart

TF-IDF0.74
LSA, 2 dimensions1
LSA, 3 dimensions1
LSA, 6 dimensionsmore dimensions did not help0.78

Printed by the application code. One query moves precision by 0.11, and the dimension was chosen on the same queries.

Anisotropy, hubness and domain shift4 minThe effect computed on synthetic vectors, how each problem shows up in real ones, and what to try.

One shared direction makes every pair look similar and creates hubs Illustration

Isotropic Gaussian
Plus one shared offset
Offset, then mean-centred

Mean cosine of random pairs: isotropic gaussian 0.00, plus one shared offset 0.52, offset, then mean-centred 0.00. With the offset every pair looks related; a threshold set on one model means nothing on another.

Hubness: the offset turns a few points into everyone's neighbour and leaves others unreachable. Largest count: 19, 84, 19; never retrieved: 0%, 7%, 0% (same order).

Seeded synthetic vectors (200 points, 48 dimensions, one shared offset of unit scale), not a real embedding model. Black bars mark the median count. The effect, not the size, is the lesson.

Mean-centering removes the shared direction and with it most of the effect (Mu et al., 2017, post-process real word vectors this way).

Geometry problems, how to see them, what to try Table

Problem	Symptom	Test	Mitigation
Anisotropy	Random pairs have high mean cosine	Mean cosine of random pairs	Mean-centering; per-model thresholds
Hubness	Same few items in many top-k lists	Count how often each item is retrieved	Centering; re-scaling scores by neighbourhood
Domain shift	Failures cluster on codes and jargon	Slice labelled queries by type	Hybrid with BM25; adapt on labelled pairs

Ethayarajh (2019) measured anisotropy in contextual representations. Hubs are a property of high-dimensional neighbours in general (Radovanovic et al., 2010), not a bug of one model. Two cheap numbers diagnose both: the mean cosine of random pairs and the largest neighbour count.

The knobs that change what similar meansadvanced3 minSimilarity function, tokenisation, weighting, dimension and threshold, with the library defaults behind today's code.

Knobs that change what "similar" means Table

Knob	Default or common choice	Trade-off
Similarity function	cosine	Dot product keeps length; which works better depends on how the model was trained
TfidfVectorizer tokens	2+ word characters, lower-cased	One-letter codes (size S, tier B) vanish from the vocabulary
TfidfVectorizer weighting	smooth idf, raw counts, L2 rows	sublinear_tf damps repeated words
TruncatedSVD	2 components, randomized, 5 iterations	Dimension is a tuned choice, as the application shows
Embedding dimension	fixed by a neural encoder (384 to 1,024 common)	Larger index and slower search vs more room
Similarity threshold	none is portable	Set per model from labelled positive and negative pairs

scikit-learn defaults checked in the installed 1.9.1; the cosine-versus-dot finding is Steck et al. (2024).

BM25, bi-encoders, cross-encoders and hybrids in production5 minThree ways to score a query against a document, their costs and failures, the retrieve-then-rerank pattern, and the versioning rule for indexes.

Three ways to score a query against a document Comparison

Criterion	BM25	Bi-encoder (dense)	Cross-encoder (reranker)
What it matches	Shared words, weighted by rarity	Query and document embedded separately	Query and document read together
Cost per query	Cheap; inverted index	One query embedding plus a vector search	One model pass per candidate pair
Index	No model; rebuild on new text only	Re-embed everything when the model changes	None: scores pairs on the fly
Typical failure	Synonyms and paraphrases	Codes, rare jargon, negation	Too slow to score a whole corpus

Reimers and Gurevych (2019) motivate bi-encoders by the cost of scoring every pair with a cross-encoder. A common pattern retrieves with BM25 or a bi-encoder and reranks the top few.

A hybrid retriever Diagram

Query
- to Dense top k
BM25 top kexact codes and rare words
- to Fuse the rankings
Dense top kparaphrases and synonyms
Fuse the rankingse.g. reciprocal rank fusion
Top results

Fusing ranks avoids comparing raw scores from two different models. Evaluate it on the same labelled queries.

What a staff-level answer addsadvanced2 minNuances and misreadings that separate an experienced answer from a textbook one.

The textbook answer and what a staff-level answer adds Comparison

Criterion	Common reading	Staff-level reading
First question	'How good are these vectors?'	Which relation were they trained to encode: topic, paraphrase, click?
The score	A probability that two items match	A ranking signal with a model-specific scale
The 2-D plot	'Similar in the plot, similar in the model'	Projections distort distances; compute neighbours in the full space
Dimensions	'More dimensions are always better'	Today's LSA got worse from 3 to 6 dimensions on the same queries
Changing the encoder	A config change	A migration: re-embed the corpus, new threshold, new evaluation

Memory anchor

A similarity score is a model's opinion about context: compare it with BM25 on your own labelled queries.

Useful for

Interpreting a similarity or nearest-neighbour result - what the number is, what it is compared with, what could distort it.
Choosing between lexical, dense and hybrid retrieval with a small labelled set before Days 41 and 48 (support search, product search, clinical-guideline lookup).
Deciding when a learned lookup table for a high-cardinality categorical feature (merchants in fraud scoring, SKUs in demand models, providers in claims) is worth its parameters.

Not suitable for

Proving that two items are "the same" - a high cosine is a ranking signal, not an identity or entailment test.
Showing that a model understands your domain; only a labelled task evaluation does that.
Drawing conclusions from a t-SNE or UMAP picture - cluster size and between-cluster distance are not reliable there.

Traps

Reading a cosine value as an absolute

A score of 0.82 sounds strong, but in a model whose random pairs already sit near 0.7 it may be unremarkable, and a threshold copied from another model silently changes precision and recall.

How to avoid it

Plot the score distributions of labelled positive and negative pairs for this model and choose the threshold from the precision or recall you need.

Judging embeddings by nearest-neighbour anecdotes

A handful of impressive neighbours and a tidy 2-D plot persuade a team; the failures live in codes, negations and rare terms that nobody queried.

How to avoid it

Score a labelled query set with a lexical baseline, slice by query type, and report counts per slice.

Senior reasoning

A senior treats an embedding as a model with a training objective, a training corpus and a failure surface. They ask what relation it was trained to encode (a shop's "bought together" is not a hospital's "same diagnosis"), compare it with BM25 on their own labelled queries, look at the score distributions before choosing thresholds, and slice the errors by query type. They consider hybrid retrieval and domain adaptation only after the slices show where the general model fails, and they plan to re-embed the corpus when the model or its preprocessing changes.

Interview question

"Users of an online shop say semantic search misses product codes and order numbers. How do you diagnose it and what do you try first?"

Outline your answer first.

Recall cards scheduled after completion

Define cosine similarity and say why a raw cosine threshold does not transfer between models.
Name the three caveats to check before trusting an embedding retrieval result.

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Indexing by latent semantic analysis — Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, Richard Harshman, 1990core
Truncated SVD of a term-document matrix as an early dense semantic representation (the LSA used in the code).
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — Nils Reimers, Iryna Gurevych, 2019core
Bi-encoder sentence embeddings compared with scoring every pair with a cross-encoder for semantic search.
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, Iryna Gurevych, 2021core
Zero-shot retrieval benchmark in which BM25 is a robust baseline and dense retrievers often underperform out of domain.
Representation Learning with Contrastive Predictive Coding — Aaron van den Oord, Yazhe Li, Oriol Vinyals, 2018optional
Introduces the InfoNCE contrastive loss used in many embedding models.
A Simple Framework for Contrastive Learning of Visual Representations — Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton, 2020optional
Contrastive learning with augmentations and many negatives for visual representations.
All-but-the-Top: Simple and Effective Postprocessing for Word Representations — Jiaqi Mu, Suma Bhat, Pramod Viswanath, 2017optional
Removing the common mean vector and a few dominant directions from word vectors improves similarity tasks (the centering fix in the geometry chapter).
Hubs in Space: Popular Nearest Neighbors in High-Dimensional Data — Milos Radovanovic, Alexandros Nanopoulos, Mirjana Ivanovic, 2010optional
Hubs appear among the nearest neighbours of many points as dimensionality grows, an inherent property of high-dimensional data.
sklearn.feature_extraction.text.TfidfVectorizer — scikit-learn developersoptional
Tokenisation and weighting defaults of the TF-IDF baseline used in the application code.
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings — Kawin Ethayarajh, 2019optional
Contextual word representations are not isotropic in any layer, which is why raw cosine values are hard to interpret.
Is Cosine-Similarity of Embeddings Really About Similarity? — Harald Steck, Chaitanya Ekanadham, Nathan Kallus, 2024optional
Cosine similarity can work better or worse than an unnormalised dot product, depending on how the embeddings were learned.
MTEB: Massive Text Embedding Benchmark — Niklas Muennighoff, Nouamane Tazi, Loic Magne, Nils Reimers, 2023optional
A benchmark across many task types in which no single embedding method dominates, supporting task-specific evaluation.
Problems With Evaluation of Word Embeddings Using Word Similarity Tasks — Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, Chris Dyer, 2016optional
Word-similarity evaluation of word vectors has several problems and is not a sustainable proxy for downstream use.
How to Use t-SNE Effectively — Martin Wattenberg, Fernanda Viegas, Ian Johnson, 2016optional
Cluster sizes and between-cluster distances in a t-SNE plot can mislead, so a 2-D picture is not evidence of separation.

Optional sections

Deep dive · 10 min

A contrastive loss by hand. Take an anchor with similarity 0.8 to its positive and 0.2 and 0.1 to two negatives, and temperature $\tau = 0.5$ . The exponentials are $e^{1.6} = 4.95$ , $e^{0.4} = 1.49$ , $e^{0.2} = 1.22$ , the softmax weight of the positive is $4.95 / 7.67 = 0.646$ and the loss is $-\ln 0.646 = 0.44$ . A lower temperature sharpens the distribution and punishes hard negatives more; the loss falls when the positive's similarity to the anchor rises or when a negative's falls, because it depends on the positive relative to the negatives.

Where the dimension comes from. In LSA the number of latent dimensions is a hyperparameter between "too few, topics blur" and "too many, back to the sparse space". In neural encoders it is fixed by the architecture (384, 768, 1,024 are common), and a smaller index is cheaper to store and search.

Categorical embeddings. An entity embedding for 1,000 merchants in a card-fraud model with $d = 16$ has 16,000 parameters, learned jointly with the task loss. It needs enough rows per level; for rare levels, share statistical strength by hashing, grouping or target-encoding as on Day 9.

Contextual versus static vectors. A static vector gives "charge" one location regardless of "card charge" or "battery charge"; a contextual encoder gives each occurrence its own vector, which helps when word sense changes the label and costs a forward pass per text.

Hands-on lab: Anisotropy and hubness on random vectors (synthetic) · 15 min

Draw 1,000 Gaussian vectors in 100 dimensions, add one shared direction to every vector, and compare the mean pairwise cosine and the neighbour-count distribution (how often each point appears in other points' 10 nearest neighbours) with and without mean-centering. This is a synthetic illustration of one cause of a narrow cone and of hubs, not a measurement of any real model. Then change the offset strength from 1.0 to 0.3 and describe how the maximum neighbour count moves.

import numpy as np
from scipy.stats import skew

rng = np.random.default_rng(38)

def knn_in_degree(X, k=10):
    """How many other points list each point among their k nearest neighbours (cosine)."""
    U = X / np.linalg.norm(X, axis=1, keepdims=True)
    S = U @ U.T
    np.fill_diagonal(S, -np.inf)
    nn = np.argsort(-S, axis=1)[:, :k]
    return np.bincount(nn.ravel(), minlength=len(X)), S

def report(name, X):
    deg, S = knn_in_degree(X)
    off = S[np.isfinite(S)]
    print(f"{name:28s} mean pair cos {off.mean():5.2f}  max in-degree {deg.max():3d}  "
          f"skew {skew(deg):4.1f}  never-retrieved {np.mean(deg == 0):.0%}")

n = 1000
low = rng.normal(size=(n, 3))
high = rng.normal(size=(n, 100))
shared = 1.0 * rng.normal(size=100)                 # one direction added to every vector
offset = rng.normal(size=(n, 100)) + shared
report("3-d Gaussian", low)
report("100-d Gaussian", high)
report("100-d with shared offset", offset)
report("same, after mean-centering", offset - offset.mean(axis=0))

Interview practice · 10 min

At a telecom operator, a product manager shows a 2-D plot of customer-ticket embeddings with three clean clusters and asks whether the embeddings "understand" complaint categories (billing, coverage, device). What do you say?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-18 · depends on numpy, scikit-learn, scipy

1 Recall · 5 min

2 Concept · 10 min

Assumptions

3 Application and workplace decision · 10 min

4 Mini-assessment · 8 min

Technical depth · optional · 33 min in total

Useful for

Not suitable for

Traps

Senior reasoning

Interview question

Recall cards scheduled after completion

Prerequisites and related concepts

References

Optional sections

Recall · 5 min

Concept · 10 min

Application and workplace decision · 10 min

Mini-assessment · 8 min