r04 results - route-day demand backtest and the forecast-versus-causal what-if

Recorded run of cd python && uv run python -m reference.r04_airline_demand_backtest.run (full mode). Every number below is copied from the files in outputs/ written by that run (metrics_by_horizon.csv, metrics_by_route.csv, slices.csv, paired_vs_baseline.csv, whatif.csv, whatif_coefficients.csv, true_price_elasticity.csv, censoring.csv, robustness_*.csv, summary.json). The data are synthetic (academy airline simulator, seed 42); the 100 EUR per passenger that appears in the what-if table is a hypothetical scale, not a fare.

Item	Value
Run date	2026-10-11
Seeds	simulator 42; LightGBM random_state 42 (robustness seeds 1, 2, 3 on 9 of the 26 origins); AutoETS is deterministic
Python / numpy / pandas / scipy	3.12.13 / 2.5.3 / 3.0.6 / 1.18.1
scikit-learn / LightGBM / statsforecast / statsmodels	1.9.1 / 4.7.0 / 2.0.1 / 0.15.0
Runtime of the recorded run	282.7 s inside run.py (4 min 46 s wall including start-up and import; single process, LightGBM n_jobs=1, AutoETS n_jobs=1; Apple M5, 10 cores, 16 GB; no other heavy job was running during this run)
Smoke mode	--smoke: 6 months of data, 3 origins, no robustness seeds, 8.6 s wall when it was timed after the final code change
Panel	12,056 route-days = 11 routes x 1,096 dates (complete grid, checked); 20,824 flight-days before aggregation
Rolling origins	26 fortnightly origins, issue dates 2023-12-19 to 2024-12-03, expanding window, horizons 1 to 28: 8,008 scored forecasts (26 x 11 routes x 28)
Target dates scored	2023-12-20 to 2024-12-31 (the first origin is in December 2023, so 12 of the first origin's target days fall in 2023)
Leakage self-checks inside the run	features at origin 2024-06-18 unchanged when every later value is replaced by 1e6 (308 rows); route-shuffle identical: True

Findings

1. The grain is clean and the features are leakage-safe. Every route has the same number of flights on every date, so the route-day grid is complete and groupby("route").shift(k) is a true k-day lag. Seasonal lags are weekday-aligned with the target date and never newer than the issue date; the 28-day trailing mean (the normaliser) ends at the issue date. Replacing every observation after an origin by 1e6 leaves the feature rows of that origin unchanged, and shuffling the route order before building features changes nothing. The unit tests also check each lag column against the documented source date on a panel whose values encode (route, day).
2. Both candidates beat seasonal naive in every horizon bucket, on every origin. Pooled over all horizons MASE is 0.988 for naive, 0.949 for seasonal naive, 0.733 for AutoETS and 0.721 for LightGBM (origin-level standard errors 0.010 to 0.024). The paired per-origin MASE difference to seasonal naive is between -0.207 and -0.246 in all six candidate-by-bucket cells, and the candidate is better on 26 of 26 origins in every cell. Success criterion (a) is met by both models. The chosen candidate by pooled MASE is LightGBM, but the margin to AutoETS is small (finding 3).
3. LightGBM and AutoETS are close; the only supported gap is at 15 to 28 days. LightGBM minus AutoETS in paired MASE is +0.006 at 1-7 days (SE 0.007, LightGBM better on 11 of 26 origins), -0.013 at 8-14 days (SE 0.013, 13 of 26) and -0.022 at 15-28 days (SE 0.009, t = -2.3, 19 of 26). On the nine-origin subset used for seeds, the gap AutoETS minus LightGBM is 0.002 / 0.017 / 0.016 MASE for the three buckets while the LightGBM seed standard deviation is about 0.002: at 1-7 days the gap is no larger than seed-to-seed variation, at 8-14 and 15-28 days it is about eight times larger. Excluding the three break routes does not change the picture (table in the robustness section).
4. The error curve is flat in the horizon. LightGBM MASE by single horizon ranges from 0.661 (h = 26) to 0.772 (h = 6) with no upward trend; the single-horizon values of the two candidates carry an origin-level standard error of 0.020 to 0.036, so differences between neighbouring horizons are mostly noise. Naive and seasonal naive coincide at h = 7, 14, 21 and 28 by construction. A flat curve is what one expects when the forecastable structure is calendar and season rather than recent level; I did not test that explanation separately.
5. Calendar information, not the algorithm, explains the holiday and event advantage. On holiday-window route-days (n = 1,243) LightGBM has MASE 0.660 against 0.772 for AutoETS, and ETS under-forecasts by 5.5% on average while LightGBM is within 0.7%. On event days (n = 132) the figures are 0.572 and 0.716. Outside holiday windows the two are level (0.732 against 0.726). LightGBM receives the holiday-window and event flags; AutoETS (one univariate model per route) does not, so this comparison mixes model class with information.
6. The structural break hurts both models and no ordering survives it. On the break routes from 2024-03-01 (n = 1,794) MASE is 0.781 (LightGBM) and 0.826 (ETS), against 0.685 and 0.693 on the other routes in the same period, and both over-forecast (+1.3% and +0.6%) because the level they learned predates the step in competitor capacity. LightGBM leads there by 0.046 MASE, but on the same routes before the step (n = 390) ETS leads by 0.040. With 390 and 1,794 scored forecasts and one break, I would not draw a conclusion about which model handles a break better.
7. Markets and routes. MASE for LightGBM / ETS is 0.718 / 0.742 domestic, 0.742 / 0.749 intl-short and 0.683 / 0.680 intl-long. By route LightGBM has the lower MASE on 8 of 11 routes and ETS on 2 (IST-DIY, IST-JFK); IST-ATH is level at three decimals. The largest route-level relative bias for LightGBM is -2.1% (IST-CDG) and no route exceeds 2.1% in absolute value.
8. Route-days at the bookable limit look biased, but that split is made on the outcome. On the 2,775 route-days with at least one flight at the bookable limit both models under-forecast (-7.7% for LightGBM) and on the other 5,233 they over-forecast (+5.1%). Because the slice is defined by a realised outcome (high bookings on the target day), a forecast that is unbiased on average must look low on one side and high on the other; the pair does not show that the model could have known at issue time that a full flight was coming. What the slice does show is the censoring itself: in the simulator's truth frame 23.0% of 2024 flight-days sit at the bookable limit and 22.3% have latent demand above it, a total of 44,369 bookings turned away in 2024, so the target systematically understates demand on those days (finding 11 shows what that does to interventions).
9. Entity-unsafe lags barely matter in this panel: there is no score inflation to report. The deliberately wrong version shifts the stacked frame, so the first 27 days of routes 2 to 11 read the tail of the previous route: 7,560 of 336,812 feature rows (2.2%) differ, all of them early-2022 training rows; no scored row is affected. MASE changes by -0.0025, -0.0044 and +0.0022 for the three buckets (mixed sign, same size as the seed range of 0.004 to 0.005). With this layout the bug is a correctness defect, not a score inflater; a panel stacked date-by-date would be far worse and was not measured.
10. The deployed forecaster cannot answer any intervention, by construction, and the code verifies that numerically. Its feature code only reads the route, the date, the target history and calendar flags. Re-running the forecaster on counterfactual inputs (capacity 180 on IST-TZX, fares x 0.95, competitor seats x 1.25) changes none of the 308 forecasts of the final origin (maximum absolute change 0.0 passengers in all three scenarios). Its answer is zero against true effects of +3,029, +37,297 and -93,669 passengers in 2024. That is the right behaviour for a forecaster, and the wrong thing to serve to a planner who asks a what-if question.
11. Capacity 160 to 180 on IST-TZX. The simulator's true effect over 2024 is +3,029 passengers on a base of 97,080 (+3.1%), although seats rise 12.5%: extra seats only help on days when bookings were at the limit (31.6% of IST-TZX flight-days in 2024). The naive pooled regression answers +8,575 (2.83 times the truth) because its capacity coefficient (0.72) is identified only by differences between routes, and routes with bigger aircraft are also different in many other ways. Capacity has zero within-route variance, so with route fixed effects it is not identified at all and the adjusted regression reports it as undefined. The driver LightGBM gives +2,032 (0.67 times): the sign is right, but with route in the model the number comes from cross-route splits and I would not trust it. An estimator that does not model the seat limit has no way to reproduce an effect that arises only on days when latent demand exceeded the old limit.
12. Price -5% on all routes. True effect +37,297 passengers (+3.7% of 1,018,336); the realised arc elasticity is -0.70 (domestic -0.84, intl-short -0.61, intl-long -0.40), smaller in magnitude than the configured route elasticities (-0.6 to -1.35); that is consistent with censoring clipping the gain on the flight-days at the limit, though I did not isolate how much of the difference is censoring. The simulator's own revenue change is -4,845,833 EUR (fares fall 5% and the extra passengers do not compensate). The pooled regression has the right sign but 0.43 of the magnitude, with a fare coefficient of -0.31; once route fixed effects, weekday, month, holiday and event flags and a trend are added the fare coefficient flips to +0.57 and the implied effect is -29,426 (-0.79 times: wrong sign). The driver LightGBM is also wrong in sign (-22,357). The reason is in the generator: the revenue-management rule sets higher fares when demand is expected or observed to be high, including a shock that no feature contains, so within a route high fares and high passenger counts occur together. The pooled model's negative sign therefore reflects differences between routes and seasons rather than a price response: it changes sign as soon as route and calendar controls are added.
13. Competitor capacity +25% on all routes. True effect -93,669 passengers (-9.2%). The naive pooled regression returns +190,615, the wrong sign (+0.77 coefficient, which is consistent with large markets attracting more competitor seats). The adjusted regression returns -99,671 (1.06 times the truth, right sign, within 6%) and by scope between 0.96 and 1.18 times; the driver LightGBM is right in sign but 0.34 of the size. So the lesson is not that regression always fails: the adjusted regression works for competitor capacity because, in the simulator, competitor seats vary within a route by independent noise plus a dated step and fares do not respond to them (see capstone/airline_sim.py), whereas fares do respond to the hidden demand shock. Whether a real competitor's capacity decision is that clean is an assumption that has to be argued, not estimated.

Backtest by horizon bucket

Pooled over 26 origins and 11 routes. MASE is scaled by the in-sample seasonal-naive MAE of the same route up to the origin; bias is forecast minus actual in passengers per route-day; the standard error is the standard deviation of the per-origin value divided by the square root of 26 (adjacent origins overlap for horizons beyond 14 days, so the standard errors are somewhat optimistic).

horizon	model	n	MAE	MASE (origin SE)	bias (forecast - actual)	relative bias
1-7	AutoETS	2,002	26.3	0.722 (0.015)	-0.68	-0.27%
1-7	LightGBM	2,002	26.5	0.729 (0.015)	-1.29	-0.51%
1-7	naive	2,002	35.2	0.968 (0.025)	-4.37	-1.73%
1-7	seasonal naive	2,002	34.3	0.938 (0.021)	-0.84	-0.33%
8-14	AutoETS	2,002	26.6	0.729 (0.014)	-0.07	-0.03%
8-14	LightGBM	2,002	25.9	0.716 (0.014)	+0.22	+0.09%
8-14	naive	2,002	35.9	0.996 (0.034)	-3.76	-1.49%
8-14	seasonal naive	2,002	35.1	0.961 (0.016)	-0.23	-0.09%
15-28	AutoETS	4,004	27.0	0.741 (0.011)	-0.21	-0.08%
15-28	LightGBM	4,004	26.2	0.719 (0.012)	-0.66	-0.26%
15-28	naive	4,004	36.1	0.995 (0.023)	-3.89	-1.54%
15-28	seasonal naive	4,004	34.5	0.948 (0.018)	-0.36	-0.14%
all	AutoETS	8,008	26.7	0.733 (0.010)	-0.29	-0.12%
all	LightGBM	8,008	26.2	0.721 (0.010)	-0.59	-0.23%
all	naive	8,008	35.8	0.988 (0.024)	-3.98	-1.57%
all	seasonal naive	8,008	34.6	0.949 (0.015)	-0.44	-0.18%

Paired per-origin MASE differences (negative favours the first model):

comparison	horizon bucket	paired MASE difference	SE over origins	t	origins where first model is better
AutoETS minus seasonal naive	1-7	-0.216	0.015	-14.5	26 of 26
AutoETS minus seasonal naive	8-14	-0.233	0.013	-18.3	26 of 26
AutoETS minus seasonal naive	15-28	-0.207	0.012	-17.9	26 of 26
LightGBM minus seasonal naive	1-7	-0.209	0.015	-13.6	26 of 26
LightGBM minus seasonal naive	8-14	-0.246	0.015	-15.8	26 of 26
LightGBM minus seasonal naive	15-28	-0.229	0.015	-15.6	26 of 26
LightGBM minus AutoETS	1-7	+0.006	0.007	+1.0	11 of 26
LightGBM minus AutoETS	8-14	-0.013	0.013	-1.0	13 of 26
LightGBM minus AutoETS	15-28	-0.022	0.009	-2.3	19 of 26

Backtest by single horizon (h = 1 to 28)

Each row pools 286 forecasts (26 origins x 11 routes). metrics_by_horizon.csv also carries WAPE, relative bias and standard errors.

h	MASE naive	MASE snaive	MASE ETS	MASE LightGBM	MAE ETS	MAE LightGBM	bias ETS	bias LightGBM
1	0.960	1.071	0.758	0.747	27.2	27.0	-1.41	-0.47
2	0.892	0.904	0.691	0.684	25.0	24.7	-0.93	+0.31
3	0.933	0.856	0.678	0.686	24.8	25.2	-6.73	-5.69
4	1.034	0.883	0.754	0.749	27.8	27.4	-0.06	-1.46
5	0.978	0.904	0.722	0.743	26.0	26.6	-3.84	-3.57
6	1.004	0.969	0.760	0.772	28.2	28.4	+4.70	+2.17
7	0.976	0.976	0.691	0.719	25.4	26.3	+3.52	-0.30
8	0.998	0.982	0.746	0.758	27.4	27.8	+5.84	+3.65
9	1.042	0.972	0.730	0.730	26.6	26.6	-2.09	-1.58
10	0.983	0.879	0.739	0.690	27.0	24.8	-2.46	-0.46
11	1.009	1.046	0.721	0.714	26.2	25.9	+1.09	+0.26
12	0.986	0.931	0.707	0.697	25.7	25.3	-1.83	-0.01
13	0.926	0.894	0.668	0.673	24.6	24.6	-2.30	-2.39
14	1.024	1.024	0.789	0.747	28.4	26.6	+1.24	+2.07
15	0.969	1.022	0.790	0.742	28.5	27.1	-1.48	-0.37
16	0.936	0.945	0.716	0.699	26.0	25.3	-0.96	+0.36
17	1.029	0.862	0.731	0.701	26.7	25.7	-5.87	-5.40
18	0.990	0.949	0.761	0.758	28.0	27.6	-0.79	-2.06
19	1.024	0.909	0.748	0.737	26.7	26.4	-2.70	-2.63
20	1.012	1.005	0.766	0.770	28.3	28.4	+4.41	+1.77
21	0.972	0.972	0.694	0.727	25.6	26.8	+3.96	+0.37
22	0.949	0.946	0.736	0.731	26.9	26.6	+5.29	+2.65
23	1.003	0.928	0.737	0.694	26.8	25.2	-2.88	-1.50
24	0.970	0.915	0.781	0.732	28.5	26.4	-1.70	-0.83
25	1.024	0.921	0.730	0.720	26.5	26.2	+0.71	-0.26
26	1.001	0.919	0.698	0.661	25.5	24.2	-1.71	-0.92
27	0.986	0.920	0.697	0.661	25.7	24.4	-1.33	-2.13
28	1.066	1.066	0.793	0.736	28.8	26.3	+2.17	+1.76

Backtest by segment

All horizons pooled; relative bias is the sum of forecast errors divided by the sum of actuals. The last two rows are defined by a realised outcome and are discussed in finding 8.

slice	n	MASE naive	MASE snaive	MASE ETS	MASE LightGBM	rel. bias ETS	rel. bias LightGBM
market: domestic	3,640	1.014	0.965	0.742	0.718	-0.1%	+0.0%
market: intl-short	2,912	1.025	0.979	0.749	0.742	+0.0%	-0.5%
market: intl-long	1,456	0.851	0.849	0.680	0.683	-0.5%	-0.6%
holiday window	1,243	1.000	0.961	0.772	0.660	-5.5%	+0.7%
outside holiday windows	6,765	0.986	0.947	0.726	0.732	+0.9%	-0.4%
destination event day	132	0.817	0.901	0.716	0.572	-7.0%	-0.3%
break routes, from 2024-03-01	1,794	1.060	1.074	0.826	0.781	+0.6%	+1.3%
break routes, before 2024-03-01	390	1.121	1.062	0.750	0.791	-0.7%	-1.2%
other routes, from 2024-03-01	4,784	0.917	0.891	0.693	0.685	-0.4%	-0.9%
route-day with a flight at the bookable limit	2,775	0.983	0.901	0.772	0.757	-8.1%	-7.7%
route-day with no flight at the limit	5,233	0.991	0.974	0.713	0.701	+5.5%	+5.1%

By route, all horizons pooled (728 forecasts per route):

route	market	MASE snaive	MASE ETS	MASE LightGBM	bias ETS	bias LightGBM	rel. bias LightGBM
IST-ADB	domestic	1.021	0.806	0.775	+2.10	+4.08	+1.0%
IST-ATH	intl-short	1.011	0.787	0.787	+0.29	-1.10	-0.9%
IST-AYT	domestic	1.070	0.811	0.775	-0.10	+2.87	+1.1%
IST-BKK	intl-long	0.847	0.680	0.679	-1.19	-1.92	-0.9%
IST-CDG	intl-short	0.870	0.693	0.690	-0.55	-2.70	-2.1%
IST-DIY	domestic	0.872	0.685	0.690	-0.47	-1.37	-1.1%
IST-ESB	domestic	0.968	0.736	0.714	-2.76	-5.22	-1.2%
IST-FRA	intl-short	1.125	0.821	0.797	+1.21	+1.14	+0.4%
IST-JFK	intl-long	0.851	0.680	0.688	-1.05	-0.79	-0.4%
IST-LHR	intl-short	0.911	0.696	0.694	-0.63	-1.50	-0.5%
IST-TZX	domestic	0.892	0.672	0.639	-0.05	-0.02	-0.0%

What-if: forecast answers against the simulator's true effect (2024, 366 days)

true_effect re-runs the simulator with the intervention and the same random draws, so the true change is exact for this world and includes the seat limit. All three interventions are applied to the whole network except the capacity change (IST-TZX only). Route demand in the simulator does not interact across routes, so a domestic-only price cut would have exactly the domestic row below. The estimators are fitted on all 2022-2024 route-days and are evaluated in sample on the 2024 route-days; they are estimators of an effect, not out-of-sample forecasts. The last column prices the error at a hypothetical 100 EUR per passenger only to give it a scale.

scenario	answer	true change	estimated change	error	estimate / truth	sign	error at 100 EUR per passenger (hypothetical)
IST-TZX capacity 160 to 180	deployed LightGBM forecaster, counterfactual inputs	+3,029	+0	-3,029	0.00	none (exactly zero)	-302,900
IST-TZX capacity 160 to 180	naive OLS (pooled)	+3,029	+8,575	+5,546	2.83	correct	+554,566
IST-TZX capacity 160 to 180	adjusted OLS (route fixed effects, calendar, trend)	+3,029	undefined	undefined	undefined	undefined	undefined
IST-TZX capacity 160 to 180	driver LightGBM (re-predicted)	+3,029	+2,032	-997	0.67	correct	-99,747
network price -5%	deployed LightGBM forecaster, counterfactual inputs	+37,297	+0	-37,297	0.00	none (exactly zero)	-3,729,700
network price -5%	naive OLS (pooled)	+37,297	+16,138	-21,159	0.43	correct	-2,115,888
network price -5%	adjusted OLS (route fixed effects, calendar, trend)	+37,297	-29,426	-66,723	-0.79	wrong	-6,672,322
network price -5%	driver LightGBM (re-predicted)	+37,297	-22,357	-59,654	-0.60	wrong	-5,965,404
competitor capacity +25%	deployed LightGBM forecaster, counterfactual inputs	-93,669	+0	+93,669	0.00	none (exactly zero)	+9,366,900
competitor capacity +25%	naive OLS (pooled)	-93,669	+190,615	+284,284	-2.04	wrong	+28,428,434
competitor capacity +25%	adjusted OLS (route fixed effects, calendar, trend)	-93,669	-99,671	-6,002	1.06	correct	-600,238
competitor capacity +25%	driver LightGBM (re-predicted)	-93,669	-31,926	+61,743	0.34	correct	+6,174,300

Simulator revenue change in 2024 (not hypothetical; the simulator's own fares): IST-TZX capacity +330,284 EUR, price -5% -4,845,833 EUR, competitor capacity +25% -18,388,790 EUR.

By scope (estimated against true change in passengers):

scenario	scope	true change	naive OLS	adjusted OLS	driver LightGBM
network price -5%	domestic	+24,121	+8,633	-15,742	-11,760
network price -5%	intl-short	+9,670	+4,855	-8,853	-6,312
network price -5%	intl-long	+3,506	+2,649	-4,831	-4,285
network price -5%	break routes (ADB, AYT, FRA)	+16,733	+5,440	-9,920	-7,370
network price -5%	other routes	+20,564	+10,698	-19,506	-14,987
competitor capacity +25%	domestic	-52,290	+101,973	-53,321	-18,097
competitor capacity +25%	intl-short	-27,472	+57,349	-29,987	-10,231
competitor capacity +25%	intl-long	-13,907	+31,293	-16,363	-3,598
competitor capacity +25%	break routes (ADB, AYT, FRA)	-34,836	+64,260	-33,601	-13,312
competitor capacity +25%	other routes	-58,833	+126,355	-66,070	-18,614

Regression coefficients (log passengers on log of the driver; HC1 standard errors):

model	term	coefficient	95% CI low	95% CI high	note
ols_naive	log avg_fare	-0.306	-0.320	-0.293
ols_naive	log capacity	0.719	0.665	0.772
ols_naive	log competitor_capacity	0.769	0.756	0.782
ols_adj	log avg_fare	0.572	0.535	0.609
ols_adj	log capacity	undefined	undefined	undefined	not identified
ols_adj	log competitor_capacity	-0.462	-0.509	-0.414

Realised effect of the -5% price change (includes the capacity cap):

scope	true change in passengers (%)	implied arc elasticity
all	3.663	-0.701
domestic	4.428	-0.845
intl-long	2.097	-0.405
intl-short	3.156	-0.606

Censoring in the simulator's truth frame, 2024 (evaluation only; never a feature): 6,954 flight-days, 23.0% at the bookable limit, 22.3% with latent demand above it, 44,369 bookings of latent demand above the limit.

route	flight_days	share at bookable limit	share with latent demand over limit	agreement	mean spill when over	total spill (bookings)
IST-ADB	1,098	0.128	0.122	0.994	20.269	2,716
IST-ATH	366	0.235	0.221	0.986	23.531	1,906
IST-AYT	732	0.094	0.093	0.999	24.323	1,654
IST-BKK	366	0.295	0.295	1.000	41.194	4,449
IST-CDG	366	0.273	0.268	0.995	26.531	2,600
IST-DIY	366	0.221	0.213	0.992	17.910	1,397
IST-ESB	1,098	0.248	0.243	0.995	25.603	6,836
IST-FRA	732	0.204	0.189	0.985	20.841	2,876
IST-JFK	366	0.331	0.331	1.000	53.496	6,473
IST-LHR	732	0.332	0.322	0.990	31.843	7,515
IST-TZX	732	0.316	0.303	0.988	26.788	5,947

Robustness

LightGBM seeds (42, 1, 2, 3) on every third origin (9 of 26); the ETS column uses the same origins:

bucket	mean MASE over seeds 42,1,2,3	sd over seeds	min	max	ETS MASE, same 9 origins	ETS minus LightGBM (seed 42)
1-7	0.7293	0.0020	0.7265	0.7310	0.7322	0.0016
8-14	0.7394	0.0022	0.7364	0.7417	0.7588	0.0171
15-28	0.7250	0.0021	0.7228	0.7270	0.7394	0.0157

Break routes excluded (8 routes, 5,824 scored forecasts), all 26 origins, MASE:

bucket	MASE naive	MASE snaive	MASE ETS	MASE LightGBM
1-7	0.933	0.885	0.696	0.707
8-14	0.968	0.927	0.701	0.696
15-28	0.965	0.899	0.709	0.694

Entity-unsafe lags (stacked-frame shift) against the safe construction, same 9 origins, LightGBM MASE:

bucket	mase_safe_lags	mase_unsafe_lags	difference
1-7	0.7306	0.7282	-0.0025
8-14	0.7417	0.7374	-0.0044
15-28	0.7237	0.7259	0.0022

What this does not prove

• The true effect exists only because the data are simulated. The conclusion (a forecaster trained on observational, confounded, censored data does not answer interventions) transfers; the numbers do not. One simulated world (seed 42) was used; the simulator seed and the generator's parameters were not varied, so the ranking of AutoETS and LightGBM and the size of each regression error are properties of this draw as well as of the method.
• The model comparison is not clean. LightGBM uses calendar and event flags that AutoETS does not, LightGBM has a fixed 120-tree setting and no tuning, and AutoETS has the default model search. A tuned ETS with regressors, or a gradient-boosting model without calendar features, would answer a different question.
• Route-day aggregation hides flight-level censoring. A route-day counts as "at the limit" when any of its flights is; a route can be well below its total cap while one flight is full. Censoring was not modelled at all, so the demand the forecasts track is boardings.
• Point forecasts only. Interval behaviour is project-022 (r05). Origins are 14 days apart and horizons run to 28, so adjacent origins share target days and the origin-level standard errors treat them as independent.
• The regression what-if answers are deliberately simple. They multiply realised passengers by exp(coefficient x change) and apply no seat limit, so even a correct elasticity would overstate capacity and price gains on censored days. The "naive" pooled regression is a strawman by design; the adjusted regression and the driver model are in-sample fits and were not tuned. A defensible causal estimate on real data would need an experiment or an identification strategy, not a better forecaster.
• The at-limit and not-at-limit slices are defined by the outcome, so their opposite biases are partly mechanical (finding 8).
• The entity-unsafe demonstration covers one bug layout (route-major stack). Its effect here is within seed noise because the contaminated rows are early training rows.
• Only 10 months of the break are scored (2024-03-01 to 2024-12-31), on three routes, so break-route results are weakly supported.
• No censoring-aware demand model (estimate latent demand on censored days, then simulate capacity) was tried; it is the natural next step named in the project's postmortem.

Reproduce

cd python && uv run python -m reference.r04_airline_demand_backtest.run          # full run, about 5 minutes
cd python && uv run python -m reference.r04_airline_demand_backtest.run --smoke  # about 10 seconds, writes to a temp dir
cd python && uv run pytest tests/test_r04_airline_demand_backtest.py             # 14 tests, about 10 seconds