Classical ML, validation, metrics, interpretation, debugging · insight-1189draft — in review
Keep a fixed reference window beside a rolling one so slow drift cannot hide
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A drift monitor compares each week's feature or score distribution with the previous week or a trailing four-week window.
Action
Compare current data with two references: a fixed one (training or last validated period) for cumulative drift, and a short rolling one for sudden shifts. Chart the distance over time, and re-baseline the fixed reference only on a recorded retraining decision.
Reason
A rolling reference measures change per step, not distance from the conditions the model was fitted on.
Boundary
Input, score and segment-mix monitoring for any deployed classical model.
Symptom or risk
Gradual change, such as a fare mix creeping upward a little each week, stays below the threshold in every comparison while the cumulative shift becomes large; the reference moves with the data.
Exceptions
Strongly seasonal data, where the fixed reference should be the same season of the last cycle.
Trade-offs
Two series double the alert surface; keep one pager and one dashboard.
Pitfalls
Re-baselining automatically each retrain without reviewing how far inputs had moved.