LLMs, GenAI, RAG, agents, evaluation · insight-0603
Compare chunk sizes at the same context token budget, not the same k
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Applied AI, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You are running a chunk-size experiment, for example 200 versus 800 tokens, and comparing recall@5 for each.
Action
Fix the context budget (say 4,000 tokens), set k per configuration so the budget is filled, and compare evidence recall, answer quality and cost under that equal budget.
Reason
Recall@k rises mechanically with the amount of text retrieved. Equalising tokens isolates whether the chunking itself puts the evidence in fewer, cleaner units.
Boundary
Any chunking, overlap or windowing ablation on day 51.
Symptom or risk
Larger chunks "win" because five 800-token chunks carry four times as much text into the prompt, so the comparison measures budget, not chunking quality.
Exceptions
When latency or cost is dominated by k (for example a per-candidate reranker), report the equal-k view as well.
Trade-offs
Small chunks raise precision but split multi-sentence evidence; large chunks keep context but dilute the prompt.
Pitfalls
Labelling relevance at chunk level: relabel evidence as document spans so the same golden set can score every chunking.