S11 · experiment simulation

Which parts of the v3 booking forecast earn their place?

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

5stages
40 minabout
0stages committed

The brief

The synthetic airline's demand team forecasts final bookings per flight-day at 21 days before departure. The lead (simulated) presents v3: "WAPE fell from 2.34% to 2.00%, a 14% improvement. It comes from the three new feature groups: calendar, 28-day lag and competitor intelligence, which cost us three months to build."

Facts about the setup (computed on flight_day and booking_curve, synthetic): train on 2022-2023, test on all 6,954 flight-days of 2024; v2 scales bookings on hand at 21 days by a per-route ratio; v3 is a LightGBM model on route, capacity and bookings at 21 days plus the three feature groups. All arms use the same split, hyperparameters and metric (WAPE, the sum of absolute errors over the sum of actual bookings).

Stage 1 of 5

You are asked to check the claim before it is written into the model card. You may run as many arms as you wish.

Which elements belong in the ablation plan? Select all that apply.

Arms that add each feature group alone to the simplest GBM, and arms that remove each group from the full v3.Ten training seeds per arm, reporting the mean and the spread across seeds rather than the result of one run.One split, one metric and one set of hyperparameters for every arm, with the v2 baseline in the same table.Re-tune each arm's hyperparameters on the 2024 test period until it looks best, then compare each arm's best result.Report the best seed for each arm, since that shows what the model can achieve when the run goes well.