LLMs, GenAI, RAG, agents, evaluation · insight-1532
Shorten embeddings only with a model that supports it, renormalise, and measure recall before taking the savings
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Vector storage and ANN memory dominate the retrieval bill, and someone proposes cutting every embedding to its first 256 dimensions.
Action
Use the model's documented dimension option or a Matryoshka-trained model, renormalise if you shorten manually, then compare recall@k and storage at two or three sizes on your labelled queries before choosing.
Reason
Matryoshka-style training packs coarse information into leading dimensions so that prefixes stay useful; vendor documentation describes a dimensions parameter and requires manual shortening to be renormalised.
Boundary
Large corpora where memory and bandwidth are the binding cost, and models with documented shortening support.
Symptom or risk
Truncating a vector from a model not trained for it, or forgetting to renormalise, lowers recall in ways that are not obvious from a smoke test.
Exceptions
Small indexes where the saving is negligible next to the risk.
Trade-offs
Fewer dimensions cut cost but usually cost some recall on hard queries; the right size is a business decision per corpus.
Pitfalls
Comparing sizes on the average only; hard or rare-term queries often lose first.
Lessons and concepts
Claims and their sources
OpenAI's embeddings guide documents a dimensions parameter for shortening embeddings, recommends setting it at creation, and says manually shortened vectors must be normalised (as read on 2026-10-12).
The Matryoshka Representation Learning paper reports encoding information at multiple granularities so one embedding can be truncated to a compute budget, with up to 14x smaller embeddings at equal ImageNet-1K accuracy (a vision result; text embedding models vary, so measure).