LLMs, GenAI, RAG, agents, evaluation · insight-0669
Expect multi-turn chat input cost to grow faster than the number of turns
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, ML engineer, Data scientist
- Industries
- telecom, cross-industry
Summary: situation, action, reason, boundary
Situation
A chat assistant resends the full conversation history on every turn, and the cost model assumes a fixed price per message.
Action
Measure input tokens per turn by conversation length; cap history, summarise older turns or keep only task state, and cache the stable prefix.
Reason
If each turn adds roughly m tokens and all history is resent, turn t carries about t·m input tokens, so total input over T turns grows roughly with T²/2 · m.
Boundary
Chat assistants, agents with long loops, support copilots.
Symptom or risk
Long conversations cost far more than short ones because each turn re-pays for all earlier turns, and latency rises with them.
Exceptions
Stateless single-turn APIs.
Trade-offs
Summarising history can drop details the user refers back to; keep key facts as structured state.
Pitfalls
Summarising untrusted content from earlier turns into instructions-like text, which can carry injected commands forward.
Example
Hypothetical: 300 tokens per turn over 20 turns means about 300 × (1 + 2 + … + 20) = 63,000 input tokens, not 20 × 300 = 6,000.