Engineering, production, governance, cost, communication · insight-1717
Generate load at a fixed arrival rate when load-testing a scoring service, not one request after another
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A load test of a model endpoint uses fixed virtual users, each sending its next request after the previous response, and reports a clean p99.
Action
Use an open model: a set arrival rate from a measured peak, independent of response times. Record latency from the scheduled send time and report the count above the objective, not only percentiles.
Reason
In a closed model a slow system receives less load. Real callers keep arriving, and the queue behind a stall is what their tail latency is made of.
Boundary
Online services with independent callers.
Symptom or risk
During a stall the test sends fewer requests, so the slow period is under-sampled and the reported tail is better than real clients would see.
Exceptions
A bounded worker pool that waits for each answer is a closed system.
Trade-offs
Open-model tests can build unbounded queues; set a cap and a time limit.
Pitfalls
Using the daily average rate instead of the busiest minute; reusing one payload.
Example
Simulated (deterministic FIFO server, 10 ms service, 50 requests per second for 60 s, one 5 s stall): open model 401 of 3,000 requests above 1 s, p99 4.7 s; one-connection closed loop 1 of 2,751 above 1 s, p99 10 ms.
Lessons and concepts
Claims and their sources
Grafana k6 documents that in a closed model the arrival rate of new iterations falls when the system under test slows down, whereas in an open model iterations start at a set rate regardless of how long earlier ones take, and links the closed-model behaviour to coordinated omission.
The wrk2 README says wrk's latency measurement under-represents slow periods because each connection waits for a response, and that wrk2 generates constant throughput and measures from when each request should have been sent.