Engineering, production, governance, cost, communication · insight-0288
Set retraining cadence from a measured decay curve and the cost of each retrain
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- retail, banking, e-commerce
Summary: situation, action, reason, boundary
Situation
A model is retrained daily because the platform makes it easy, and nobody has checked whether daily retrains help.
Action
Backtest models trained at different ages against the same later period to estimate how fast accuracy decays, then choose the cadence where the value of recovered accuracy exceeds compute, validation and release cost.
Reason
Decay rates differ by problem, so the cadence should come from evidence about this model.
Boundary
Scheduled retraining for forecasting, ranking, risk and fraud models.
Symptom or risk
Frequent retrains add cost, version churn and risk of shipping a bad model, possibly for no accuracy gain.
Exceptions
Adversarial domains with fast-changing behaviour may need triggered retraining in addition to schedules.
Trade-offs
Longer cadences save cost but react slower to sudden changes; keep a triggered path for shocks.
Pitfalls
Measuring decay on a period that included a known break.