Engineering, production, governance, cost, communication · insight-1279
Set the dynamic batching delay from the latency objective, then measure what it buys
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- e-commerce, saas, cross-industry
Summary: situation, action, reason, boundary
Situation
A GPU-backed ranking model serves single requests at 15% utilisation, and a proposal is to batch requests for up to 100 ms.
Action
Take the latency budget left after feature fetch and network, increase the maximum queue delay stepwise until the 99th percentile approaches that budget, record throughput and cost per thousand predictions at each step, and keep the best point.
Reason
Batch size and queue delay trade latency for throughput, and the gain depends on traffic shape, so it has to be measured.
Boundary
Neural and GPU-served models, and CPU models with vectorised inference.
Symptom or risk
Batching raises throughput but adds queueing delay to every request; an unbounded or arbitrary delay breaks the latency objective at low traffic, where batches rarely fill.
Exceptions
Where requests are already batched upstream or latency is not constrained, use the largest batch.
Trade-offs
Higher utilisation lowers cost but increases the tail and makes latency depend on traffic level.
Pitfalls
Tuning on a synthetic load generator whose arrival pattern is smoother than production.
Lessons and concepts
Claims and their sources
NVIDIA Triton's dynamic batcher holds requests up to a configurable maximum queue delay to form larger batches, and the documentation advises raising the delay until latency reaches the budget to see how much throughput it buys.