LLMs, GenAI, RAG, agents, evaluation · insight-0659
Give each request one deadline and pass the remaining time to every downstream call
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Retrieval, reranking, the model call and tool calls each have their own fixed timeout, or none.
Action
Set an end-to-end deadline per request type, give each stage the remaining budget, cancel work that cannot finish in time, and stream long generations instead of waiting for one large response.
Reason
Google's SRE guidance recommends deadlines with propagation so servers stop work that can no longer complete; Anthropic's API docs recommend streaming or batches for long-running requests.
Boundary
Interactive RAG and agent requests with several network hops.
Symptom or risk
The user gives up after 10 seconds while backend work continues for a minute; long generations hit idle-connection drops and fail without a response.
Exceptions
Offline batch jobs, which need job-level timeouts instead.
Trade-offs
Strict deadlines turn slow-but-correct responses into errors; return a partial or fallback answer where safe.
Pitfalls
Retrying after the deadline has already passed.
Lessons and concepts
Claims and their sources
Google's SRE guidance recommends setting deadlines on requests and propagating the remaining deadline to downstream calls.
Anthropic's API documentation recommends the streaming or batch APIs for long-running requests and warns that idle connections can be dropped.