Time series and forecasting · insight-1212
Split squared forecast error into bias and variance across origins before choosing a fix
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A model's RMSE is above target and the team is deciding between a bias correction, more regularisation and an ensemble.
Action
Over the rolling origins compute the mean error (bias) and the variance of the error, and compare squared bias with the error variance, per horizon and per segment. Correct systematic offsets first; use ensembling or smoothing for the variance part.
Reason
Mean squared error equals squared mean error plus error variance. The two parts respond to different remedies and have different business costs.
Boundary
Diagnosing any recurring forecast, especially after a structural change or a model swap.
Symptom or risk
Without knowing which part dominates, the team can spend weeks reducing variance when the error is a stable offset, or add a level shift to a model whose error is noise.
Exceptions
Origins that are strongly autocorrelated make the variance estimate optimistic; use enough distinct periods.
Trade-offs
Bias correction estimated on the same window it is judged on will look better than it is.
Pitfalls
Reading a near-zero overall bias as no bias when segments offset each other.
Example
Hypothetical: bias +6 units and error sd 10 give MSE 136; bias is 26% of it, so removing it cuts RMSE from 11.7 to 10.