S20 · experiment simulation

A directional golden case fails on a good demand forecaster

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
40 minabout
0stages committed

The brief

The synthetic airline's demand model predicts passengers per flight-day from route, capacity, weekday, month, day of year, holiday, event, competitor capacity and, in version v2, the average fare. It is trained on 2022-2023 and evaluated on 2024 (6,954 flight-days) with a gradient-boosted tree model (seed fixed).

Revenue management maintains a behavioural golden case: "With everything else held fixed, a 10% higher fare must not increase predicted passengers." It was written from the belief that demand falls when price rises. In this synthetic world the simulator's true price elasticities are negative (-0.6 to -1.35 by route), but fares are set higher when demand is expected or observed to be high. Numbers below were recomputed from the course database and the same model code.

Stage 1 of 4

Model	2024 WAPE	Prediction after +10% fare (all 6,954 flight-days)
v1, no fare feature	14.58%	not applicable
v2, with average fare	14.46%	rose on 90.9% of days, fell on 4.8%; mean change +3.4%

The directional case fails on v2. An engineer says: "The model learned the wrong sign. It has a bug."

How do you read the failure?

A defect: a correct demand model cannot show demand rising with price, so look for a sign error or a leaked target in the fare feature and fix the training pipeline, then re-run both the holdout and the directional case.The test is stale and should be deleted, since the model beats the baseline on the holdout and the holdout is the metric the business signed off.Competitors' prices are missing, which causes the wrong sign; adding competitor price and sold-out indicators to the features will restore the expected direction.The model fits the observed association: fares were set higher on days of higher expected demand. It forecasts well but is not a price-response model, so the case asks something it was never trained to answer.