Engineering, production, governance, cost, communication · insight-0267
Page on model impact symptoms, and send per-feature drift tests to a dashboard
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A drift monitor runs a statistical test on 300 features daily and pages on every p-value below 0.05.
Action
Page only on urgent, actionable symptoms (prediction volume, null rates, decision rates, matured performance); put feature-level drift on a dashboard ranked by effect size and feature importance; review thresholds after every false alarm.
Reason
Pages should require urgent human judgment, and noisy alerts get ignored and can hide real outages; many simultaneous tests also inflate false positives.
Boundary
Model monitoring stacks with many features or many models.
Symptom or risk
Many tests on large samples flag tiny, harmless shifts every day, people stop reading the alerts, and a real failure is missed.
Exceptions
Safety-critical inputs (a sensor feeding an automated control) may justify paging on a single feature.
Trade-offs
Fewer pages mean some issues wait for daily review.
Pitfalls
Using p-values on millions of rows, where every difference is significant.
Lessons and concepts
Claims and their sources
Google SRE guidance says pages should be urgent, actionable and need human judgment, and noisy alerts get ignored and can hide real outages.