Engineering, production, governance, cost, communication · insight-1268
Aggregate latency from histogram buckets, never by averaging per-instance percentiles
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A dashboard shows model p99 latency as the mean of the p99 values reported by 12 replicas, and it looks fine while users complain.
Action
Export latency as histograms with bucket boundaries at the objective, sum bucket counts across replicas, compute quantiles from the summed buckets, and break out by model version and route.
Reason
Precomputed quantiles are not aggregatable across instances, while bucket counts can be summed before the quantile is estimated.
Boundary
Replicated model servers, batch job durations, feature-store call latencies.
Symptom or risk
The average of percentiles is not the percentile of the combined traffic, so a single slow replica or a heavy-tailed minority hides inside a healthy-looking number.
Exceptions
A single instance can use its own summary quantile.
Trade-offs
Bucket resolution limits the accuracy of estimated quantiles, and boundaries need revisiting when the objective changes.
Pitfalls
Coarse buckets around the objective, which make a service look worse or better than it is.
Lessons and concepts
Claims and their sources
The Prometheus documentation says that averaging pre-computed summary quantiles across instances is statistically meaningless, while histogram bucket counts can be aggregated and passed to histogram_quantile, and that classic histograms need a bucket boundary at the objective threshold.