LLMs, GenAI, RAG, agents, evaluation · insight-1530
Price a full re-index from chunk size, overlap and header tokens before fixing those settings
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A team picks 20% chunk overlap and a generated context header per chunk, and plans to re-embed the corpus whenever the model or chunker changes.
Action
Compute embedded tokens as chunks times (chunk size plus header size), where chunks is source tokens divided by stride; multiply by your price and throughput, and review the figure as a recurring cost.
Reason
Overlap and per-chunk headers multiply the text sent to the embedding model, and the multiplier is paid again at every re-index, not only the first.
Boundary
Planning chunk parameters, embedding-model upgrades and generated contextual headers.
Symptom or risk
Embedded tokens grow well beyond source tokens, so each re-index costs and takes longer than budgeted and teams start avoiding model upgrades.
Exceptions
Small corpora where the whole re-index costs less than a day of engineering time.
Trade-offs
Overlap and headers can lift recall; the point is to buy them knowingly, not to avoid them.
Pitfalls
Forgetting the generation cost of LLM-written headers, which is separate from embedding cost.
Example
Hypothetical: 10M source tokens, 500-token chunks with 100 overlap (stride 400) give 25,000 chunks and 12.5M embedded tokens; a 60-token header lifts that to 25,000 x 560 = 14.0M, a 1.4x multiplier on the source tokens (1.25x without the header).