LLMs, GenAI, RAG, agents, evaluation · insight-1520
Allocate the context window as a ledger with per-section ceilings and a fixed eviction order
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A booking assistant concatenates system rules, a user profile, conversation history, retrieved fare rules and tool results, and requests intermittently overflow or lose the rules.
Action
Give every section a token ceiling from measured p95 sizes, reserve output tokens first, define an eviction order (for example old tool results, then older turns, never the rules), enforce it in code and log each eviction.
Reason
The context window is one shared, fixed budget; without an allocation policy, truncation is decided by concatenation order rather than by importance.
Boundary
Assistants and agents that assemble prompts from several variable-size sources.
Symptom or risk
Whichever section grows last pushes out another, usually through blind truncation from one end, so the fare rule the answer needs disappears without a log entry.
Exceptions
Prompts far below the limit with stable section sizes.
Trade-offs
Ceilings need re-measuring when content changes, and summarising evicted content costs extra calls.
Pitfalls
Measuring in characters or words instead of the deployed tokenizer's tokens, especially for non-English text.