LLMs, GenAI, RAG, agents, evaluation · insight-1640
Forecast token spend from the per-request length distribution, not from the median request
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, Data scientist, ML engineer
- Industries
- retail, aviation
Summary: situation, action, reason, boundary
Situation
A cost forecast for a new assistant multiplies expected requests by the length of a typical prompt seen in the pilot.
Action
Segment pilot traffic by request type, forecast requests per segment with growth ranges, use each segment's mean tokens for input, output and cached input separately, and publish low, expected and high scenarios.
Reason
Total spend is a sum over requests, so it follows the mean, which a heavy tail pulls above the median.
Boundary
Budget requests and capacity plans for GenAI features.
Symptom or risk
A heavy tail of long requests, which the median ignores, makes real spend much higher than the forecast and the budget is blown in the first month.
Exceptions
Fixed-length batch jobs, where one measured request length is enough.
Trade-offs
Pilot traffic may differ from launch traffic; update monthly.
Pitfalls
Ignoring retries, tool-call loops and conversation growth, which raise tokens per task.
Example
Hypothetical: 70% of requests have 800 input tokens, 25% 2,500 and 5% 12,000, so the mean is 1,785, not the 800 median. At 100,000 requests a day and 357 a day, not $160.