Engineering, production, governance, cost, communication · insight-0291
Monitor default and fallback rates to catch silent model failures
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
An upstream join breaks, a key feature becomes null, and the pipeline fills it with the training median.
Action
Count, per feature and per request, how often imputation, defaults or fallback branches are used; alert when those rates exceed their historical range; and expose them in the decision log.
Reason
Imputation hides missing inputs from the model and from error monitoring, so its rate is the signal that something upstream broke.
Boundary
Any pipeline with imputation, default values or fallback logic.
Symptom or risk
The service returns valid-looking predictions with no errors, so standard monitoring stays green while decisions degrade.
Exceptions
Features that are legitimately often missing need seasonal or segment baselines for their null rate.
Trade-offs
More metrics to maintain; limit to features with high importance.
Pitfalls
Imputing in the feature store so the serving layer never sees that a value was missing.