Engineering, production, governance, cost, communication · insight-1277

Add jitter, a cap and a retry budget when many batch workers call a model service

Situation
A nightly scoring job runs 200 workers against the model endpoint, which has a 30-second brownout, and every worker retries on the same fixed schedule.
Action
Use randomized exponential backoff with a maximum delay, a per-worker and per-job retry budget, a deadline for the whole request, no retries on permanent errors, and a limit on concurrent workers.
Reason
Retries multiply load on a struggling dependency, and uncoordinated fixed schedules synchronize clients; randomization and budgets bound the amplification.
Boundary
Batch scoring, feature backfills and any fan-out client of a shared model service.

Symptom or risk

Retries arrive in synchronized waves, the brownout becomes an outage and the job takes hours longer than the failure itself.

A single sequential client with low volume can use simple capped retries.

Budgets cause some work to fail fast; pair with a resumable job that skips completed partitions.

Retrying at every layer, which multiplies attempts.

The SRE book chapter on cascading failures recommends randomized exponential backoff, limited retries per request and a retry budget, warns that retries at several layers multiply, and says permanent errors should not be retried.
Site Reliability Engineering, Chapter 22: Addressing Cascading Failures