LLMs, GenAI, RAG, agents, evaluation · insight-1637
Isolate long-prompt traffic or chunk prefill when document requests stall chat token streams
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- ML engineer
- Industries
- saas, public-sector
Summary: situation, action, reason, boundary
Situation
A shared deployment serves short chat turns and occasional 50,000-token document analyses, and chat users report streams that freeze for seconds.
Action
Enable chunked prefill with a token budget per iteration, or place long-document requests on a separate pool, and report p95 inter-token latency by prompt-length class, not one blended number.
Reason
The Sarathi-Serve authors describe prefill as high-latency and compute-saturating and decode as fast but under-using compute, so mixing them hurts one or the other; chunked prefills with stall-free scheduling raised serving capacity under tail-latency limits in their tests.
Boundary
Self-hosted serving with mixed prompt lengths and streaming users.
Symptom or risk
Long prefill iterations occupy the GPU, so ongoing decodes stall and inter-token latency spikes for unrelated users.
Exceptions
Uniformly short prompts, or hosted APIs where you cannot configure scheduling.
Trade-offs
Chunking lengthens the long request's own time to first token; a separate pool costs idle capacity.
Pitfalls
Tuning on average latency, which hides the stalls that users notice.
Lessons and concepts
Claims and their sources
Sarathi-Serve (a preprint) reports that chunked prefills with stall-free scheduling gave 2.6x higher serving capacity than vLLM for Mistral-7B on one A100 and up to 3.7x for Yi-34B on two A100s, and up to 5.6x for Falcon-180B with pipeline parallelism, under tail-latency constraints on the setups tested.