LLMs, GenAI, RAG, agents, evaluation · insight-1538
Give the retrieval path a per-stage latency budget, and gate the stage that dominates
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Time to first token is over target, and the team debates faster vector search while the path also contains an LLM query rewrite and a reranker.
Action
Record p50 and p95 per stage, run independent branches in parallel, and gate expensive optional stages such as rewriting behind a cheap check that skips them for self-contained queries.
Reason
Sequential stages add, parallel branches take the maximum, and an LLM call usually dwarfs index lookups; measuring each stage shows where a change can matter.
Boundary
Interactive RAG with rewrite, hybrid retrieval and rerank stages.
Symptom or risk
Effort goes into the 25 ms stage while a 600 ms model call sits in front of retrieval on every request.
Exceptions
Offline batch answering, where throughput and cost matter more than latency.
Trade-offs
Gating needs its own accuracy check, and skipped rewrites will miss the cases the gate misjudges.
Pitfalls
Optimising median latency while the p95 is set by rare slow reranker or cold-cache calls.
Example
Hypothetical medians in ms: rewrite 600, embed 40, ANN 25, BM25 20, rerank 200 sum to 885, of which rewrite is 68%. Parallelising ANN and BM25 saves 20 ms (865); skipping rewrite for half the queries saves about 300 ms on average.