Classical ML, validation, metrics, interpretation, debugging · insight-0174
Plot residuals in time order even for models that are not forecasts
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- aviation, retail
Summary: situation, action, reason, boundary
Situation
A cross-sectional regression is trained on data collected over months, such as flight-level revenue or store-level sales.
Action
Plot held-out residuals against date and check autocorrelation; if structured, add time-aware features, validate out of time, or model the dependence.
Reason
Independent errors are assumed by standard inference and by random splits; time-ordered patterns break both.
Boundary
Any regression on data collected over time, including panel data.
Symptom or risk
Residuals drift or cluster in time; the model misses a regime change and standard errors are understated.
Exceptions
Experiments with randomized timing and stable conditions rarely show such patterns.
Trade-offs
Time features can fix drift in training but extrapolate poorly; monitor them.
Pitfalls
Looking only at the residual histogram, which is blind to order.
Example
Synthetic airline data: competitor capacity on three routes steps up 25% on 2024-03-01, so a model without that feature shows residuals on those routes shifting after that date.
Lessons and concepts
Claims and their sources
Correlated error terms make estimated standard errors underestimate true standard errors; plotting residuals against time reveals tracking (ISL section 3.3.3).