Neural networks, deep learning, NLP · insight-1475
Backtest sequence models in free-running mode, not with teacher-forced inputs
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- aviation, energy
Summary: situation, action, reason, boundary
Situation
A recurrent forecaster predicts the next 14 days of bookings, and validation loss is computed by feeding the true previous values as inputs.
Action
Evaluate by rolling the model on its own outputs over the real horizon, report error by step, and if the gap is large try scheduled sampling or a direct multi-horizon output head.
Reason
Bengio et al. describe the mismatch between training on true previous tokens and inference on the model's own outputs, note that errors can compound, and report improvements from gradually shifting training to the model's own outputs.
Boundary
Autoregressive RNNs and Transformers for multi-step forecasts and generation.
Symptom or risk
In production the model consumes its own predictions, so errors compound with the step, and the teacher-forced score overstates multi-step accuracy.
Exceptions
One-step-ahead use.
Trade-offs
Free-running evaluation is slower; scheduled sampling adds a schedule to tune.
Pitfalls
Reporting only step-one error; leaking future covariates into the free-running rollout.
Lessons and concepts
Claims and their sources
Bengio et al. (arXiv preprint) describe the mismatch between training on true previous tokens and inference on the model's own outputs, note that errors can compound, and report improvements from scheduled sampling on several tasks.