LLMs, GenAI, RAG, agents, evaluation · insight-0665
Keep a tested fallback model path for provider overloads and outages
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- aviation, telecom
Summary: situation, action, reason, boundary
Situation
A customer-facing assistant depends on one hosted model, and the only failure plan is "retry".
Action
Define a fallback chain per feature (another model or region, a cached or templated answer, or a handoff to people), keep fallback prompts tuned and evaluated on the golden set, and trigger the switch on error rates rather than single failures.
Reason
Hosted APIs return transient overload and server errors; Anthropic documents a 529 overloaded error that can occur under high traffic across all users. Prompts and schemas rarely transfer unchanged between models, so the fallback must be tested in advance.
Boundary
Assistants with availability targets, peak-season traffic (for example airline disruption days), or contractual SLAs.
Symptom or risk
During a provider overload or regional incident, retries exhaust the latency budget and every user sees errors, or an untested switch to another model breaks output formats.
Exceptions
Internal tools where a short outage is acceptable.
Trade-offs
Each fallback adds evaluation and maintenance work, and may answer with lower quality; say so in the interface when degraded.
Pitfalls
Failing over to a model whose data-processing terms or region the compliance review never approved.
Lessons and concepts
Claims and their sources
Anthropic's API returns a 529 overloaded_error when the API is temporarily overloaded, which can happen during high traffic across all users.