r13 results - final bookings per departure from booking-curve snapshots (project-016)

Recorded run of cd python && uv run python -m reference.r13_booking_curve_final_bookings.run (full mode). Every number below is copied from the files in outputs/ written by that run (summary.json, data_checks.csv, metrics_by_horizon.csv, criterion1_paired.csv, per_origin_wape.csv, slices.csv, intervals.csv, interval_coverage_by_slice.csv, decision_costs.csv, decision_sensitivity.csv, criterion4_decision.csv, robustness.csv, leak_controls.csv, censoring_note.csv, success_criteria.csv, report.md). The data are synthetic (academy airline simulator, seed 42); fares and costs in the decision analysis are hypothetical.

Item	Value
Run date	2026-10-12
Seeds	database 42; LightGBM random_state=42, deterministic=True, n_jobs=1 (a second LightGBM seed, 7, is a robustness row)
Python / numpy / pandas / scipy / scikit-learn / lightgbm	3.12.13 / 2.5.3 / 3.0.6 / 1.18.1 / 1.9.1 / 4.7.0
Machine	Apple M5, macOS arm64, CPU only
Runtime of the recorded run	166.4 s (shared laptop; other full runs during development took 100 to 120 s). --smoke: 8 s.
Database	airline_sample.sqlite, sha256 a364e86b2c27990888ace9486e90e53ca5df4535816f68586a47dcf7447eb284; flight_day 20,824, booking_curve 270,712, calendar 1,096 rows (match the project record)
Evaluation	12 monthly origins in 2024 (expanding window), 19 flights, issue horizons d = 30, 14, 7; 6,384 / 6,688 / 6,821 evaluated forecasts
Tuning	4-cell LightGBM grid on the 12 monthly origins of 2023, using only departures dated before 2024-01-01; frozen in outputs/frozen_config.yaml before 2024 was scored
Determinism	the repeat run inside robustness.csv is bit-identical (determinism_max_abs_diff = 0.0)
Tests	uv run pytest tests/test_r13_booking_curve_final_bookings.py (see the end of this file)

Findings

1. The data satisfy the five quality rules of the record with zero violations. 13 snapshots for each of 20,824 departures, no duplicate keys among 270,712 rows, no decreasing cumulative curve among 249,888 consecutive snapshot pairs, the d = 0 snapshot equals bookings_final on all 20,824 departures, and each of the 19 flights has 1,096 dates. 19.3% of departures (4,013) end at the bookable limit floor(1.06 x capacity), so the target is truncated bookings, not demand (26.3% on the 280-seat flights, 21.1% on 160 seats, 17.4% on 180 seats).

2. Additive pickup is the better baseline at every horizon, and the candidates beat it by a wide margin. Pooled 2024 WAPE:

   d	seasonal naive	additive pickup	regression pickup	LightGBM
   30	0.180	0.084	0.028	0.026
   14	0.172	0.044	0.016	0.013
   7	0.147	0.025	0.013	0.009

   MASE (scaled by the in-sample seasonal-naive error) of LightGBM is 0.138, 0.066 and 0.048. Against additive pickup, LightGBM lowers WAPE by 68.4%, 71.7% and 64.2%; the mean per-origin difference is 34.2, 42.9 and 61.2 standard errors (12 of 12 origins better at every horizon), so success criterion 1 passes with a large margin. LightGBM was also the candidate chosen on 2023 at all three horizons.

3. Most of the gain comes from a one-line model; LightGBM adds a small, consistent increment. Regression pickup (final on cum at d, one line per market) already reaches 0.028, 0.016 and 0.013. LightGBM improves on it by 0.0014, 0.0031 and 0.0036 WAPE points on the per-origin mean (SE 0.0002 each, 12 of 12 origins better), a relative gain of 5%, 19% and 28%. The extra gain is real in this data but small in absolute seats: mean absolute error 4.30, 2.04 and 1.47 seats for LightGBM against 4.53, 2.55 and 2.06 for regression pickup.

4. No pre-declared slice is worse than the baseline. Relative to additive pickup the LightGBM change in WAPE ranges from -65.8% to -81.3% (d = 30), -64.7% to -84.9% (d = 14) and -55.2% to -81.8% (d = 7) over the six criterion-2 slices (three markets, holiday window or event, ex post at the limit, IST-ADB/AYT/FRA after 2024-03-01). On the three break routes after the competitor step LightGBM WAPE is 0.030, 0.014 and 0.010 (all departures: 0.026, 0.013, 0.009), so the step is visible as a mild degradation, not as a failure. The slice "at the bookable limit" is defined by the outcome and is reported because the project record names it, but it cannot be used at issue time.

5. The 80% intervals reach the target coverage overall and over-cover where bookings are capped. Coverage of the shipped LightGBM intervals is 79.8%, 81.0% and 80.9% at d = 30, 14 and 7; mean width is 13.0, 6.4 and 4.6 seats. By slice the range is 75.7% to 92.9% (d = 30), 77.4% to 92.8% (d = 14) and 76.8% to 94.6% (d = 7); the highest value is always the "at the bookable limit" slice (93% to 95%), where the forecast is capped and the residual is one-sided. Intervals are relative residual quantiles from matured forecasts by horizon and market, rebuilt for every origin from departures dated before that origin, plus the 2023 pool.

6. The first decision rule failed; a second, calibrated rule was added afterwards and is cheaper than both references. Costs are in hypothetical currency per departure, with the fares of decision_fares_hypothetical.csv (spoilage = L fare, displacement = H - L fare; domestic 79.0 and 24.7).

   d	never close L	close when point >= capacity	rule 4a: P(limit) from residual quantiles	rule 4b: calibrated P(limit given f / limit bin)	share closed by 4b	oracle
   30	10.25	15.07	10.25	7.62	13.2%	0
   14	10.08	13.90	10.08	5.83	20.2%	0
   7	10.01	13.57	10.01	4.97	22.2%	0

   Rule 4a never closes a class, so it equals "never close L": pooled relative residuals are roughly symmetric and about half of them are non-negative, so at a forecast equal to the limit the estimated probability is about 0.5, below the break-even probability L / H (0.76 for domestic). Rule 4b estimates P(final bookings reach the limit) in six fixed bins of forecast / limit (0.9, 0.95, 0.98, 0.99, 0.999) from matured forecasts only, and closes L when the estimate exceeds the break-even probability. In the 2024 forecasts of d = 14 the bins reach the limit in 0.0%, 0.3%, 34.9%, 70.0%, 86.9% and 94.1% of cases from the lowest bins upward.

7. The cost ranking depends on the cost ratio. For d = 14, with the displacement cost multiplied by 0.25, 0.5, 1, 2, 4, 8, 16: never close costs 2.52, 5.04, 10.08, 20.16, 40.32, 80.65, 161.3; rule 4b costs 2.47, 4.24, 5.83, 7.90, 9.50, 12.48, 13.36 and closes 3.8%, 10.9%, 20.2%, 23.8%, 24.6%, 25.4%, 30.3% of departures; rule 4a only starts closing at a multiplier of 4 (cost 25.97). At 0.25 the improvement of 4b over never-close is 2%, which is small enough that I would not rely on it. Scaling the residual spread by 0.5 to 2 left 4a unchanged (it closes nothing at multiplier 1).

8. The controls behave as expected, with one that shows nothing. Adding the d = 0 snapshot (c0 / cum_d) drops WAPE to 0.0007, 0.0005 and 0.0004, which is what a leak looks like. Adding the realised fare of the departure changes WAPE from 0.0265 to 0.0257 (d = 30), 0.0126 to 0.0125 (d = 14) and not at all at d = 7: in this simulator the fare adds almost nothing beyond the booking curve, so the control does not demonstrate a leak and the exclusion is justified by availability (the fare is an outcome of the departure), not by a measured effect. The entity-unsafe positional shift reads another flight on 665, 399 and 266 rows (the first weeks of each flight) and leaves WAPE unchanged to three decimals, so it is a correctness defect that the unit test catches and the accuracy table does not.

9. Robustness. An alternative training window starting 2023-03-01, excluding the three break routes from training, a different LightGBM seed and +-10% on the pace feature at prediction time all keep LightGBM ahead of the better baseline at every horizon (WAPE 0.026 to 0.027, 0.013 to 0.014, 0.009; maximum absolute forecast difference from the base run 0.9 to 4.3 seats). The four tuning cells differ by at most 0.0002 WAPE in 2023, so the grid is within noise and the "tuned" configuration is a default in practice.

10. Censoring. On departures that ended at the limit, the mean unconstrained demand (simulator truth, used for this table only) is 226 against 198 observed bookings, and the shipped forecast is 12.6% to 13.6% below unconstrained demand while only 0.4% to 1.5% below observed bookings. On the other departures the forecast is within 0.5% of both. The model forecasts what the booking system will record, not what customers wanted.

Success criteria (record of project-016)

#	Criterion	Result	Detail
1	beat the better baseline by more than 2 SE at every horizon	met	d = 30: 34.2 SE; d = 14: 42.9 SE; d = 7: 61.2 SE
2	no pre-declared slice worse than the baseline by more than 5% WAPE	met	worst slice -65.8%, -64.7%, -55.2% (lower is better)
3	80% interval coverage between 75% and 85% at every horizon	met	79.8%, 81.0%, 80.9%
4a	the first written rule (residual-quantile P(limit)) is cheaper than never-close and point >= capacity	not met	it equals never-close at all three horizons
4b	a second rule, added after 4a was seen to fail (calibrated P(limit given forecast / limit bin)), is cheaper than both	met, post hoc	7.62, 5.83, 4.97 against 10.25, 10.08, 10.01 and 15.07, 13.90, 13.57

Criterion 4 is met only by a rule that was designed after the first rule failed on the 2024 numbers. Its bins were fixed in advance of seeing its own costs and nothing was tuned on 2024, but the choice to try it was informed by 2024 results, so the 4b row is evidence that a calibrated rule is feasible, not an out-of-sample confirmation.

What this does not prove

• Synthetic data. The simulator draws final bookings from a smooth booking curve with a known noise structure; a WAPE of 1% to 3% is a property of that generator (every snapshot is a noisy share of the final count), not a benchmark for real airline data. Pickup methods are known to be strong in real revenue-management data too, but real curves, group bookings, cancellations and schedule changes are absent here.
• The decision cost is a stylised proxy. One marginal seat per departure, spoilage equal to the full L fare, hypothetical fares from 2022-2023 averages of the synthetic carrier. Closing a class changes later bookings; the backtest forecasts were produced under historical RM behaviour and do not model that feedback. A nested-class control would change both the rule and the cost.
• One year, one seed, 12 origins. The standard errors are over 12 monthly origins, which are not independent (overlapping seasonal patterns, the same 19 flights). The interval estimates do not include seed or data-generation uncertainty.
• Calendar facts are assumed known in advance (holidays and announced destination events). The competitor columns have no observation timestamp and are excluded; with timestamps they could be tested as features.
• Tuning and candidate choice are tiny. Four cells that differ by 0.0002 WAPE, one candidate choice per horizon. The first 7, 14 or 30 days of 2024 are not evaluated (issue dates in 2023; evaluation_coverage.csv: 570, 266 and 133 departures).
• Rule 4b was chosen after seeing 2024. See above. The residual-quantile rule 4a and the calibrated rule 4b both ignore that probabilities are estimated from the same 19 flights.

Reproduce

cd python
uv run python -m reference.r13_booking_curve_final_bookings.run          # about 2 to 3 minutes, writes outputs/
uv run python -m reference.r13_booking_curve_final_bookings.run --smoke  # about 10 seconds, temp directory
uv run pytest tests/test_r13_booking_curve_final_bookings.py