LLMs, GenAI, RAG, agents, evaluation · insight-0660
Retry LLM calls at exactly one layer, with jittered backoff and respect for retry-after
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Your code wraps the provider SDK call in its own retry loop, and a job queue also retries failed jobs.
Action
Decide which layer owns retries (often the SDK), configure its maximum, use randomised exponential backoff, honour retry-after, and make other layers fail fast with the error type.
Reason
Google's SRE book shows retries multiplying across layers (three retries at three layers can mean 64 attempts). Anthropic's SDK already retries transient errors twice by default with backoff, so an extra loop stacks on top.
Boundary
Any service calling hosted model APIs.
Symptom or risk
During an overload, each layer multiplies attempts, adding load exactly when the provider is struggling and burning your rate limit.
Exceptions
Idempotent, cheap reads during brief network blips may tolerate a second layer with a strict budget.
Trade-offs
Fewer retries surface more errors to users; pair with fallbacks such as a secondary model.
Pitfalls
Retrying spend-limit or invalid-request errors that will never succeed.
Lessons and concepts
Claims and their sources
Retries at multiple layers multiply; Google's SRE guidance recommends randomised exponential backoff, per-request retry limits and retry budgets.
Anthropic's official SDK retries transient failures with exponential backoff, twice by default, honouring the retry-after header.