LLMs, GenAI, RAG, agents, evaluation · insight-1645
Size the fallback model for the whole failover load and shed low-priority traffic first
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- ML engineer
- Industries
- aviation, logistics
Summary: situation, action, reason, boundary
Situation
The primary provider degrades and the fallback chain sends 100% of traffic to a secondary model whose quota was set for occasional use.
Action
Check the fallback's rate limits against peak failover traffic, set priority classes (interactive over batch and evaluation), shed or defer low-priority requests first, cap retries, and rehearse the failover under load before an incident.
Reason
The SRE book describes load shedding, failing early and cheaply, and the way naive retries and stacked layers multiply attempts and destabilise systems near their limits.
Boundary
Multi-provider or multi-model chains for user-facing features.
Symptom or risk
The secondary is overwhelmed, retries amplify the load, and both paths fail together, including the interactive traffic that mattered most.
Exceptions
Internal tools with a human-handoff fallback and low volume.
Trade-offs
Reserving fallback capacity costs money when idle, and shedding means some users get a degraded response.
Pitfalls
Testing failover with a trickle of requests, which never reaches the quota that breaks in production.
Lessons and concepts
Claims and their sources
The SRE book's cascading-failures chapter describes load shedding and graceful degradation, advises failing early and cheaply, and warns that retries at several layers multiply attempts and can destabilise a system near its limit.