S11 · experiment simulation

Which part of forecast v2 actually helped?

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

5stages
40 minabout
0stages committed

The brief

A demand-forecast model for the synthetic airline predicts final bookings 21 days before departure. A colleague (simulated) reports: "v2 cuts WAPE from 12.4% to 11.4%." Recall from the course world that competitor capacity on IST-ADB, IST-AYT and IST-FRA stepped up by 25% on 2024-03-01. All results in this scenario are simulated.

Stage 1 of 5

Details of the claim:

12.4% is v1 (call it A) from last year's model card, evaluated on a 2023 holdout.
11.4% is v2 (call it B) on a June 2024 holdout, the best of 5 random seeds.
B differs from A in four ways: competitor-fare features, an event calendar, target-encoded route, and tuned hyperparameters.

What is the main problem with the claim?

The numbers come from different periods, B's is the best of five seeds, and four changes were made at once.WAPE is the wrong metric for this problem: it is dominated by large routes, so a one-point change says little about the small routes.Nothing; a one-point gain is large.