Engineering, production, governance, cost, communication · insight-0268
Confirm that drift hurts performance before retraining on it
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- banking, insurance, retail
Summary: situation, action, reason, boundary
Situation
A drift report shows the age distribution of applicants shifted, and the team schedules an urgent retrain.
Action
Check performance on the shifted segment with matured labels or a sliced backtest, estimate the expected impact, and retrain only when evidence shows degradation or the new region is poorly covered by training data.
Reason
Drift is a change in data, not proof of a change in error; the action should follow the impact.
Boundary
Scheduled and triggered retraining decisions.
Symptom or risk
Covariate shift in a region where the model is accurate may not reduce performance, and retraining on a small recent window can make things worse.
Exceptions
Concept drift with a clear external cause (new regulation, product change) may justify retraining before labels confirm it.
Trade-offs
Waiting for evidence delays fixes; retraining by reflex adds churn and risk.
Pitfalls
Retraining on data that includes the model's own influenced decisions without noticing.