Engineering, production, governance, cost, communication · insight-0259
Design a degraded mode for when upstream features are late or missing
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- aviation, retail, banking
Summary: situation, action, reason, boundary
Situation
The nightly feature pipeline fails, and the pricing model must still publish recommendations by 06:00.
Action
Decide in advance, per failure type, between serving with last-known features and a flag, switching to a simpler fallback model or rule, or abstaining and keeping yesterday's decisions; test each path.
Reason
Pre-agreed fallbacks keep decisions safe and explainable during incidents instead of improvised.
Boundary
Batch and online models feeding operational decisions with deadlines.
Symptom or risk
Without a plan, the service either crashes, serves silently with defaults, or someone runs ad hoc fixes under pressure.
Exceptions
Low-stakes recommendations can simply be skipped when inputs are missing.
Trade-offs
Maintaining a fallback model costs effort; a static rule is often enough.
Pitfalls
Fallbacks that are never exercised and fail when needed.