Skip to content
ML Academy

Introduced on day 54

Time to first token (TTFT)

The delay between sending a request and receiving the first output token; it includes queueing and processing the whole input, so it grows with prompt length.

In detail

Total response time is roughly TTFT plus (output tokens minus one) times the time per output token. Streaming lets users see text from TTFT onward but does not shorten the total. Prompt caching and shorter inputs lower TTFT; shorter outputs and faster decoding lower the generation part.