Feature engineering and preprocessing · insight-0117
Monitor missing-rate shifts because a changed form or feed silently rewrites learned missingness
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- banking, insurance
Summary: situation, action, reason, boundary
Situation
A model uses missing indicators or native NaN routing, and an upstream form, API or ETL job changes.
Action
Track per-feature missing rates in training and serving, alert on large changes, and retrain or remove the indicator when the collection process changes.
Reason
Missingness patterns reflect a process, not a law of nature; models that learn from them inherit the process's instability.
Boundary
Production models fed by forms, partner feeds, CRM systems or joined tables.
Symptom or risk
A field that was optional becomes mandatory or a join starts failing; the indicator's meaning flips and predictions shift without errors.
Exceptions
Features with stable, structural missingness (a field undefined for one product) need less monitoring.
Trade-offs
More monitors mean more alerts to triage; prioritize features with high importance.
Pitfalls
Comparing serving missing rates with a training snapshot taken after cleaning that serving never sees.