LLMs, GenAI, RAG, agents, evaluation · insight-0674
Measure time-to-first-token and per-token generation time separately
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- telecom, cross-industry
Summary: situation, action, reason, boundary
Situation
A latency SLO is defined only on total response time for a streamed chat assistant.
Action
Record time to first token (queue plus prompt processing) and time per output token (generation speed) at p50 and p95; cut prompt length or cache prefixes to lower the first, shorten outputs or change model to lower the second.
Reason
OpenTelemetry's GenAI metrics define both measures separately, and Anthropic documents that prompt cache hits lower time-to-first-token, which shows the two phases respond to different levers.
Boundary
Streaming chat, voice assistants, agent steps that block on model calls.
Symptom or risk
Optimisation targets the wrong phase: long prompts inflate the wait before anything appears, while long answers inflate the rest, and one number hides which dominates.
Exceptions
Batch or non-streaming use, where total time is the relevant measure.
Trade-offs
Faster time-to-first-token from smaller prompts can cost answer quality if needed context is removed.
Pitfalls
Measuring latency on a warm, idle test environment instead of at production concurrency.
Lessons and concepts
Claims and their sources
OpenTelemetry's GenAI metric conventions (status Development) define time to first token (including queue and prefill time) and time per output token after the first.
Prompt cache hits reduce processing time and time-to-first-token for long prompts.