Engineering, production, governance, cost, communication · insight-1280
Budget the tail latency of every dependency when one request fans out to many lookups
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- ML engineer
- Industries
- e-commerce, banking, cross-industry
Summary: situation, action, reason, boundary
Situation
Each prediction needs 20 parallel feature lookups, and every store has a good median and a p99 of 80 ms.
Action
Compute the fan-out's probability of at least one slow call, give each dependency a p99 budget and timeout, reduce the number of lookups per request, and define a fallback value for a late feature.
Reason
Latency tails get harder to control as systems scale up, and brief slowdowns that are harmless in one component can dominate at the request level.
Boundary
Online models that assemble features from several stores or services.
Symptom or risk
The request waits for the slowest lookup, so a rare slow call on any one of 20 becomes common at the request level and the overall p99 is far worse than any component's.
Exceptions
Where all features come from a single co-located store with one read, there is no fan-out to budget.
Trade-offs
Timeouts with fallbacks trade completeness for latency and can add skew when a feature is often defaulted.
Pitfalls
Reporting only the model compute time while the lookups dominate the request.
Example
Hypothetical: if each lookup exceeds its p99 once in 100 calls, the chance that at least one of 20 does is 1 - 0.99^20, about 18%.
Lessons and concepts
Claims and their sources
Dean and Barroso argue that latency tails become harder to control as systems grow and that brief slowdowns harmless at small scale can dominate performance at large scale, and propose latency tail-tolerant techniques.