r02 results - calibrated passenger no-show probabilities for overbooking and reconfirmation

Recorded run of cd python && uv run python -m reference.r02_noshow_classifier.run (full mode, five generator seeds, primary seed 20261011). Every number below is copied from outputs/metrics.json, outputs/results_tables.md, outputs/slices.csv, outputs/seed_study.csv and outputs/model_card.md, written by that run. The data are synthetic (academy airline simulator schedule plus a passenger-level generator with a known logistic no-show process); all costs are hypothetical teaching values for the fictional carrier ZX.

Item	Value
Run date	2026-10-11
Seeds	generator seeds 20261011 to 20261015 (the simulator seed equals the generator seed); primary 20261011
Python / numpy / pandas / scipy / scikit-learn	3.12.13 / 2.5.3 / 3.0.6 / 1.18.1 / 1.9.1
Runtime of the recorded run	179.2 s wall for all five seeds (180.9 s by time; 494 s user CPU), 29 to 42 s per seed, on a 10-core laptop shared with other jobs. An earlier run of the same code, with identical results, took 204.9 s at a higher machine load.
Primary seed data	480,230 bookings on 3,257 departures (22% of the simulated departures); train 2022-07-01 to 2023-12-31 = 326,762 rows (no-show 6.31%); calibration 2024-01-01 to 2024-02-29 = 35,902 (6.37%); test 2024-03-01 to 2024-08-31 = 117,566 on 770 flights (7.33%)
Baseline	route x fare-level rate table, 33 cells, none below the pseudo-count of 50, shrunk toward the market rate

Findings

1. The learned models beat the rate table on log loss by 5.7 to 6.3% in every seed, which is above the 5% bar, and they close most of the available gap. Primary-seed test log loss: rate table 0.2584, logistic regression 0.2431, gradient boosting 0.2437 (raw) and 0.2437 (sigmoid-calibrated), isotonic-calibrated 0.2446, simulator true probabilities (oracle) 0.2416. The selected model (calibrated boosting on this seed) improves on the rate table by 5.69% and closes 87% of the rate-table-to-oracle gap; the oracle itself improves on the rate table by 6.52%. Across the five seeds the selected model's improvement is 5.69 to 6.27% (mean 5.93%, sd 0.24 pp).

2. Ranking quality is modest, calibration is already good. Test ROC-AUC 0.604 (rate table), 0.699 (logistic), 0.696 (boosting), 0.704 (oracle); PR-AUC 0.100, 0.183, 0.182, 0.191 against a base rate of 7.3%. The rate table's ECE (10 equal-count bins) is already 0.0079, logistic regression 0.0050, raw boosting 0.0059, the selected sigmoid-calibrated boosting 0.0055 and isotonic 0.0068. Calibration of the selected model is within the 0.01 bar in all five seeds (0.0055 to 0.0088). Raw boosting was already inside the bar (0.0059 to 0.0098), so the calibration step bought little here; sigmoid was chosen in all five seeds by cross-fit log loss on the calibration slice.

3. Logistic regression and boosting are tied; the family choice is a coin flip. On the calibration slice the cross-fit log loss of boosting+sigmoid (0.21557) is 0.0002 below logistic regression (0.21580), so boosting was selected on the primary seed; across seeds the selected family is boosting in 2 and logistic regression in 3. Test log loss differs by at most 0.0006 between the two on any seed. This is consistent with the generator being logistic (it favours the logistic model), and says nothing about real passengers.

4. Gap to the simulator's true probabilities: the overall and every-slice bars are NOT met. Mean absolute gap 0.0116 for the selected model on the primary seed (bar 0.01), 0.0097 for logistic regression, 0.0112 for raw boosting and 0.0333 for the rate table. Across seeds the selected model's gap is 0.0096 to 0.0116 and is within the bar in one seed of five. Slices above 0.02 on the primary seed: one, IST-ADB (gap 0.0229, mean predicted 0.0703 against true 0.0927). The three break routes after 2024-03-01 have gaps of 0.0229 (IST-ADB), 0.0171 (IST-AYT) and 0.0161 (IST-FRA); together they have a gap of 0.0195 and a mean bias of -0.0188 (predicted 0.0670, true 0.0859). The other routes together have a gap of 0.0080 and a bias of +0.0036, inside the bar. The cause is the +0.30 logit shift that the generator applies to those routes at the test start and that no feature can encode; the models were trained before it. Across seeds 1 to 3 slices exceed 0.02 (their identity is recorded for the primary seed only). No slice has fewer than 100 events.

5. Outside the route slices the largest gaps are 0.0145 to 0.0170. Lead time 60 days or more 0.0170 (bias +0.0009), the corporate channel 0.0160 (bias -0.0096), fare level H 0.0151 and departures within one day of a holiday 0.0145. All are inside the 0.02 bar.

6. Overbooking: the model-based limits cost less than the rate-table limits in every seed. Expected cost per flight under the hypothetical EUR 600 denied boarding and EUR 150 empty seat (ratio 4), evaluated with the true probabilities over the 770 test flights:

   Policy	Cost per flight (EUR)	Mean limit above capacity (seats)
   no overbooking	2,066	0.0
   rate table	927	10.1
   logistic regression	825	10.6
   boosting, raw	828	10.4
   boosting, calibrated (selected)	835	11.0
   oracle (true probabilities)	716	11.7

   The selected policy saves 91.9 EUR per flight against the rate table (approximate SE 9.1 over flights; cheaper on 52.9% of flights and costlier on 25.8%) and 43.5% of the rate-table-to-oracle gap. Over five seeds the mean cost is 928 (rate table), 830 (logistic), 832 (raw boosting), 830 (selected), 716 (oracle) and 2,062 (no overbooking). Savings against the rate table at denied-to-empty ratios 1 / 2 / 4 / 8 / 16 are 86.6 / 90.6 / 91.9 / 90.0 / 89.2 EUR per flight; the advantage does not disappear over that range. Every flight receives some overbooking (the limit is above capacity on 100% of flights), because the expected no-show count is far above one seat.

7. Calibration did not make the decision cheaper. On the primary seed the calibrated boosting policy (835) costs more than raw boosting (828) and logistic regression (825) although their test log losses differ by 0.0006 or less; policy cost responds to small shifts in the predicted no-show rate (the sigmoid, fitted on a January-February slice, moved the mean prediction from 0.0674 to 0.0701). One seed cannot say whether this is systematic: on seed 20261015 the selected policy (822) is cheaper than raw boosting (835), on seed 20261014 it is the logistic model and costs 813 against 811 for raw boosting.

8. Reconfirmation threshold from costs, not from F1. The threshold is contact cost / (contact effect x freed-seat value) = 2 / (0.30 x 60) = 0.111. Under the true probabilities, contacting the bookings the selected model scores at or above 0.111 gives a net benefit of 27.1 EUR per flight with 23.8 contacts per flight; the rate table, whose probabilities rarely exceed 0.111, contacts 6.1 per flight and nets 2.6 EUR; the oracle nets 29.0 EUR; contacting everyone loses 103.3 EUR per flight. The F1-optimal threshold chosen on the calibration slice is 0.119 and nets 26.7 EUR per flight, only 1.5% below the cost-based rule here, so on this synthetic data the two rules nearly coincide; the cost-based one is the one that can be re-derived when costs change.

9. The contact burden is uneven by channel. Contact rates of the selected model: agency 23.0%, corporate 20.6%, direct 9.4%, against observed no-show rates of 8.7%, 9.3% and 5.8%. The synthetic data have no protected attributes; this only shows the mechanism a real review would look for.

10. The leakage trap is large. Adding checked_in_online (known only from T-1) to boosting moves test log loss from 0.2437 to 0.1466 (40% lower), ROC-AUC from 0.696 to 0.934 and PR-AUC from 0.182 to 0.494. The metric jump is real in the table and unobtainable at T-3, which is why the point-in-time audit rejects the column.

11. Point-in-time and split audits pass. Every one of the 11 features has availability at or before T-3; the stored prior_noshows_12m equals its recomputation from the history table for all 480,230 bookings and every history trip dates before the booking; no PNR spans flights or splits; train, calibration and test departures are strictly ordered.

12. Seed study.

    Seed	Booking rows	Test no-show rate	LL rate table	LL logistic	LL boosting (raw)	LL selected (family)	Improvement vs rate table	ECE selected	Gap to true p, selected	Slices > 0.02 gap	Cost/flight rate table	Cost selected	Cost oracle
    20261011	480,230	7.33%	0.2584	0.2431	0.2437	0.2437 (GBM+sigmoid)	5.7%	0.0055	0.0116	1	927	835	716
    20261012	483,406	7.47%	0.2623	0.2464	0.2465	0.2464 (logistic)	6.0%	0.0077	0.0106	3	951	841	719
    20261013	477,578	7.53%	0.2639	0.2488	0.2488	0.2488 (logistic)	5.7%	0.0088	0.0104	3	928	838	713
    20261014	464,613	7.31%	0.2583	0.2431	0.2429	0.2431 (logistic)	5.9%	0.0056	0.0096	1	896	813	712
    20261015	491,987	7.46%	0.2616	0.2454	0.2454	0.2452 (GBM+sigmoid)	6.3%	0.0061	0.0108	1	939	822	721

Success criteria of the project record

Criterion	Result	Met
1a. Test log loss at least 5% below the rate table	5.69% on the primary seed (5.69% to 6.27% across seeds)	yes
1b. ECE (10 bins) at most 0.01	0.0055 (0.0055 to 0.0088 across seeds)	yes
2a. Mean absolute gap to true probabilities at most 0.01 overall	0.0116 on the primary seed (0.0096 to 0.0116; within the bar in 1 of 5 seeds)	no
2b. Gap at most 0.02 in every pre-registered slice	one slice fails on the primary seed (IST-ADB, 0.0229); 1 to 3 across seeds. Reported as a failure	no
3. Policy cost lower than the baseline's, across 5 seeds	lower in all 5 (813 to 841 against 896 to 951 EUR per flight)	yes
4. Reconfirmation threshold from a stated cost ratio	0.111 = 2 / (0.30 x 60)	yes

Criteria 2a and 2b are reported as not met. The slice-level evidence points at the unmodelled break routes (finding 4), not at a defect in the pipeline, but the criterion is stated on the whole test period and is not met as written.

What this does not prove

• It does not show that gradient boosting or logistic regression would be well calibrated on real passengers. The generator is logistic and its coefficients are hypothetical; the near-tie between the two families is a property of that choice.
• The gap-to-truth criteria can be evaluated only because the simulator's probabilities are known; real data offer only realised outcomes, where reliability curves and slice-wise calibration are the available evidence.
• The overbooking costs are expected costs under the true probabilities with hypothetical EUR figures, single leg and single class. They assume each flight sells up to the chosen limit, give extra bookings the flight's mean risk, and treat the T-3 booking-curve count as live bookings. The Poisson-binomial ignores within-PNR correlation (half of the multi-passenger PNRs share one draw in the generator), so it understates the variance of shows and the limits are likely a little aggressive. No cancellations after T-3 are modelled.
• The test period starts on the break date by design, so the break-route slices measure drift that no model fitted on earlier data could capture, not model quality; whether a monitoring plan would catch it was not simulated.
• Seeds regenerate the schedule, bookings and labels but share the same logistic coefficients, so the seed spread describes sampling variation of one process, not variation across processes.
• The sigmoid-versus-isotonic and logistic-versus-boosting choices rest on a 35,902-row calibration slice and differ by 0.0002 to 0.0013 in log loss; the selection is not statistically meaningful.
• Splits are by departure date without a 3-day purge for label delay; this cannot leak anything in the synthetic generator but would matter on real data.
• The 5 to 6% log-loss improvement is a property of the effect sizes set in the generator (the oracle gets 6.5%); the size of the improvement says nothing about real revenue-management data.
• The runtime depends on free cores (179 s and 205 s for the same code on the same machine).

Reproduce

cd python && uv run python -m reference.r02_noshow_classifier.run
cd python && uv run pytest tests/test_r02_noshow_classifier.py