Engineering, production, governance, cost, communication · insight-0304
Limit and explain forecast churn between model versions
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer, Data analyst
- Industries
- aviation, retail, logistics
Summary: situation, action, reason, boundary
Situation
A new forecasting model version changes next month's plan by 15% for some routes, though inputs barely changed.
Action
Measure the change in forecasts between the old and new version for the same origin, set an acceptable churn level, explain large changes by segment, and phase in or smooth changes when accuracy gains are small.
Reason
Users act on forecasts over time, so stability is part of usefulness, not only accuracy.
Boundary
Planning forecasts, risk scores, rankings used repeatedly by people.
Symptom or risk
Planners lose trust when forecasts jump between releases without a reason, and they start overriding the model.
Exceptions
When the new model fixes a known error, a large change is correct; communicate it as a correction.
Trade-offs
Smoothing delays accuracy improvements.
Pitfalls
Releasing a new model in the middle of a planning cycle.