LLMs, GenAI, RAG, agents, evaluation · insight-1529
Assert chunk token length against the embedding model's input limit at ingest, because truncation is silent
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Chunks come from a splitter counting characters or a different tokenizer than the embedding model, and some policy sections are very long.
Action
Count tokens with the embedding model's own tokenizer, fail or re-split any chunk over the limit, and log the share of chunks within 10% of it.
Reason
Some embedding models truncate rather than raise; the E5 card states long texts are truncated to at most 512 tokens, so the vector describes only the head of the chunk.
Boundary
Indexes built from structure-based chunks, tables, long clauses or concatenated metadata headers.
Symptom or risk
The model embeds only the first part of an over-long chunk, so facts in its tail are unsearchable although the full text is stored and shown to the generator.
Exceptions
Models or APIs that return an error for over-length input; those fail loudly, but the test is still cheap.
Trade-offs
Sub-splitting breaks structure-based chunks; keep the parent id so the full section can be restored.
Pitfalls
Adding a context header after the length check, which pushes chunks over the limit again.
Lessons and concepts
Claims and their sources
The e5-base-v2 model card states that long texts are truncated to at most 512 tokens.