LLMs, GenAI, RAG, agents, evaluation · insight-0675
Plan model API capacity in tokens per minute, and count what your provider counts
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- e-commerce, aviation
Summary: situation, action, reason, boundary
Situation
Capacity planning for a launch estimates requests per minute only, using average prompt sizes.
Action
Forecast peak requests, input tokens and output tokens per minute per model from logged distributions, check which of them binds, ramp traffic gradually, and use caching where the provider excludes cached reads from limits.
Reason
Anthropic measures limits in requests, input tokens and output tokens per minute with token-bucket replenishment, so bursts can trip limits; for most of its models, cached input reads do not count toward input-token limits.
Boundary
Launch planning, marketing-driven spikes, batch backfills sharing limits.
Symptom or risk
Traffic stays under the request limit but hits the input-token or output-token limit at peak, because long-context requests cluster.
Exceptions
Self-hosted models, where GPU memory and batch size bound throughput instead.
Trade-offs
Higher limits may require a higher spend tier or reserved capacity.
Pitfalls
Planning from averages; peak-minute token volume is what triggers rejections.
Lessons and concepts
Claims and their sources
Anthropic's rate limits are measured in requests per minute, input tokens per minute and output tokens per minute, use a token-bucket algorithm, and for most models exclude cache reads from input-token limits.