S05 · adr simulation

ADR: how a forecast should be refreshed after the market changes

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
40 minabout
0stages committed

The brief

A seven-day-ahead passengers-per-flight model has run in production since January 2024 at a synthetic airline (simulated company). It was trained on 2022-2023. On 2024-03-01 a competitor raised capacity by 25% on IST-ADB, IST-AYT and IST-FRA (a documented fact of the course's simulator). The team is deciding how the model should be refreshed. One ML engineer is on call for the platform.

The comparison below was run on the course's synthetic flight table: a gradient-boosted model with calendar, seats, route and two lag features, evaluated from 2024-03-01 to 2024-12-31 (5,814 flight-days; one run, seed 0). Bias is (sum of forecast - sum of actual) / sum of actual.

Stage 1 of 4

Policy	WAPE all	Bias all	WAPE affected routes	Bias affected routes
Seasonal baseline (mean of last four same weekdays)	14.4%	+0.0%	17.0%	+0.5%
Model frozen at the end of 2023	14.1%	-2.3%	15.8%	+2.8%
Same model retrained every 30 days on the trailing 365 days	13.8%	-0.1%	16.0%	+4.2%
Frozen model times a trailing 28-day mean actual/forecast ratio per flight (lagged seven days)	13.6%	-0.1%	15.8%	+0.3%

Affected routes are IST-ADB, IST-AYT and IST-FRA (2,142 flight-days); the other eight routes have 3,672.

Which conclusions does the table support? Select all that apply.

The ratio correction has the lowest WAPE in this run, with bias near zero overall and on affected routes, and needs no retraining job.Monthly retraining on 365 trailing days did not repair the affected routes: bias grew from +2.8% to +4.2%, since most of the window predates the break.The WAPE differences between policies are under one point and come from a single run on synthetic data, so the ranking should not be over-read.Never refreshing the model is clearly unacceptable, because its WAPE is the worst of the four policies and nothing else matters.The break is invisible in the metrics, so monitoring for it is not worth building, and retraining on a schedule is enough.