S22 · defense simulation

A forecasting headline under committee questioning

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
40 minabout
0stages committed

The brief

Day 60. A simulated committee (a staff data scientist, an ML engineering lead and a hiring manager) reviews a project in the pack: a passenger forecaster on the synthetic airline sample (19 flights, 2023-07-01 to 2024-12-31, 10,450 flight-days, mean 146.9 passengers per flight-day). The model is a gradient-boosted tree model on lagged same-weekday values, the four-week mean, calendar and capacity features. The README headline reads: "ML model reaches 13.9% WAPE versus 14.6% for the seasonal baseline".

The baseline is the mean of the same flight's four previous same weekdays. The committee has the README and this evidence table, which you produced and recomputed from the course database.

Stage 1 of 4

Evaluation design (same 10,450 flight-days)	Model WAPE	Baseline WAPE	Difference
Random 5-fold split	13.86%	14.55%	0.69 points
Rolling origin: six quarterly cutoffs, train on everything before	14.25%	14.55%	0.30 points

Staff data scientist: "Your README says 13.9%. Which evaluation produced that, and why is that the right headline?"

How do you answer?

The 13.9% is from random 5-fold, which is the standard way to estimate error and uses all the data for both training and testing, so it is the most precise estimate of the model's accuracy.Both are valid and I would show them side by side, with the average of 14.1% as the headline, so readers can see the model is robust to how the data are split.Random 5-fold is fine for a model that uses only lagged features because lags are already in the past; leakage is only a problem for models that use future values directly.The 13.9% came from random folds, which do not match deployment. The headline should be the rolling-origin 14.25% against 14.55%, a gain of 0.30 points, and I should correct the README now.