S09 · defense simulation

Defend a booking forecast under challenge

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
30 minabout
0stages committed

The brief

You built a gradient-boosted model that forecasts final bookings per flight-day, 21 days before departure, for the synthetic airline in the course database (flight_day, booking_curve, 2022-2024). Revenue management uses the forecast to set fare-class availability three weeks out.

Your written report says:

Evaluation: rolling-origin backtest, expanding window, six monthly cutoffs in 2024, horizon fixed at 21 days.
Features: bookings on hand at 21 days out, same-flight lags from prior weeks, weekday, holiday and event flags, route, and competitor_price.
Metric: MAPE per flight-day, averaged. Headline: MAPE 23.5%, versus 33.0% for a seasonal-naive pickup baseline.

A review panel will question you: a staff data scientist, a senior ML engineer and the head of revenue management. All three are simulated, and so are their opinions and the numbers they quote. This is the written alternative to a spoken defense: answer as you would in the room, then read the reveal.

Stage 1 of 4

Simulated staff DS: "I re-ran your model with shuffled 5-fold cross-validation over all flight-days and got WAPE 9.8%. Your rolling-origin backtest gives WAPE 16.2%. Why should we believe the worse number?"

Simulated diagnostic you already have: residuals of the same flight on consecutive weeks are positively correlated (lag-1 weekly autocorrelation 0.41 on the 2024 holdout).

How do you answer the split challenge?

Switch to shuffled 5-fold. More folds use more data per estimate, every route-week gets tested, and the result no longer depends on which calendar period happened to be held out, so the estimate is simply more reliable and one number can stand for the whole year.Keep rolling-origin. Shuffled folds train on later departures of the same flights, and autocorrelated residuals show neighbouring weeks share information. The 9.8% measures gap-filling, not forecasting 21 days ahead; I will show both, labelled.Report the average of the two numbers as a compromise.Keep rolling-origin because that is the standard for time series, and the forecasting literature does not use shuffled folds.