r10 results - data validation gate, drift monitoring with delayed labels, retrain decision

Recorded run of cd python && uv run python -m reference.r10_drift_monitoring_pipeline.run (full mode). Every number below is copied from the files in outputs/ written by that run (summary.json, validation_fault_coverage.csv, validation_injected_faults.csv, false_alarm_rates.csv, detection_delays.csv, detection_summary.csv, alert_precision_vs_event_log.csv, threshold_grid_2023H1.csv, psi_sample_size_note.csv, drift_alerts.csv, matured_label_wape_by_month.csv, retrain_options_holdout.csv, retrain_revisit.csv, robustness_*.csv, decision_record.md, monitoring_report.md). The data are synthetic (academy airline simulator, database seed 42).

Item	Value
Run date	2026-10-11 (started 23:36 local time, UTC+3)
Seeds	database 42; fault injection 20241011; LightGBM random_state=42, deterministic=True, n_jobs=4
Python / numpy / pandas / scipy / lightgbm	3.12.13 / 2.5.3 / 3.0.6 / 1.18.1 / 4.7.0
Runtime of the recorded run	90.8 s (92.6 s wall including interpreter start; shared laptop, CPU only). Other full runs of the same pipeline during development took 60 to 114 s. --smoke: 26.4 s wall.
Database	airline_sample.sqlite, sha256 a364e86b2c27990888ace9486e90e53ca5df4535816f68586a47dcf7447eb284; flight_day 20,824, booking_curve 270,712, flight_day_raw 13,887, calendar 1,096 rows (match the project record)
Replay	1,096 daily batches, 2022-01-01 to 2024-12-31; drift and performance monitors evaluated from 2023-01-01
Production model v1	LightGBM per issue horizon (d = 30 and d = 7) on booking-curve snapshots, trained on departures up to 2022-12-31 only
Simulated faults injected in 2024	late feed 2024-07-16; renamed column 2024-08-13; fare and competitor price in cents 2024-12-21 to 2024-12-23
Determinism	two full runs of this code produced identical output files except runtime_seconds
Tests	uv run pytest tests/test_r10_drift_monitoring_pipeline.py: 17 passed in 25.6 s

Findings

1. Every documented raw fault is caught by an executable check on the batch of the day it belongs to, and the gate never blocked a clean batch. flight_day_raw against its clean twin: 110 duplicated keys (half with a 2% late revenue update), 281 rows with missing fare and revenue, 6 load factors above 1, 2 negative bookings, 1 passenger count inflated tenfold, 124 lower-case route codes and 93 missing flight-days. Recall is 100% for each type, with 0 extra rows flagged by the specific check. The three simulated 2024 faults (5 batch-days) were all blocked on their first day by the checks freshness.newest_date_equals_batch_date and bc.freshness.snapshot_date_equals_batch_date (late feed), schema.columns_present (renamed column) and the market range checks on fare and competitor price (cents). The naive pipeline consumed all 5 of those batches as delivered; the designed pipeline scored 0 of them. Over all 1,096 batches exactly 5 were blocked, all of them injected. The three booking-curve structure checks on the full table report 0 failures in 20,824 departures.
2. The common default (PSI >= 0.10 or KS p < 0.05 against all of 2022) is permanently on in this feed. Over 2023 the naive monitor had 0.152 alert episodes per route-feature stream-month, which looks small, but 0.965 of all stream-days sat inside an open alert and all 44 streams alerted at least once. The reason is sample size, not drift: a 28-day window holds a median of 54 to 56 observations per route against 678 to 730 in the 2022 reference, so the PSI expected under no shift at all is 0.173 to 0.179, above the 0.10 threshold. The raw PSI was at least 0.10 on 87% (competitor capacity) to 99% (competitor price) of stream-days. Counting episodes alone hides this because the alert never closes.
3. A seasonal-matched reference with a sample-size-adjusted PSI and a KS condition is quiet enough, and the fixed 2022 reference cannot be made quiet by thresholds alone. The chosen rule (same 28 days one year earlier, padded by 7 days; adjusted PSI >= 0.05 AND KS p < 0.001 on 3 consecutive days; closes after 7 quiet days) was selected on 2023 H1 as the most sensitive grid cell within the budget (1 episode per feature per month counted over all 11 routes, and at most 10% of stream-days open). It produced 0.0235 episodes per stream-month in H1 (0.26 per feature-month), 0.0340 in the held-out H2 (0.37 per feature-month) and 0.0293 over 2023 (0.32 per feature-month), with 1.5% of stream-days in an open alert and 11 of 44 streams alerting at all. Success criterion 2 is met under the strict reading (one feature counted over all routes). The same tuning on the fixed 2022 reference found no grid cell within budget; its most conservative cell (adjusted PSI 0.30, KS 0.001) still produced 0.154 episodes per stream-month in 2023 with 17.2% of stream-days open. The seasonal design cannot run in the first 13 months: it is defined only from 2023-02-03, so January 2023 is not monitored by it.
4. The 2024-03-01 competitor-capacity step is flagged by input monitoring within 14 days on all three routes, but only through the cause variable. The seasonal-matched rule raised on competitor_capacity 6 days after the step on IST-AYT, 8 days on IST-ADB and 10 days on IST-FRA (0 earlier episodes in the 60 days before). The naive monitor raised nothing new on competitor_capacity (0 of 3): all three streams were already inside an open alert. Under the chosen strict rule none of the three booking-curve streams (cum30_norm) or forecast streams (forecast_norm) alerted in the window (0 of 6), and competitor_price alerted on 1 of 3 (IST-ADB, 53 days, not a variable the step changes directly). The sensitive grid cells do see the demand side: with KS p < 0.05 or 0.01 and an adjusted PSI threshold of 0.20 or lower, all 9 break streams (3 routes by 3 features) alert, with the competitor-capacity delay at most 6 to 8 days, but at 1.3 to 2.5 episodes per feature-month on 2023 H1 and up to 12.6% of stream-days open (threshold_grid_2023H1.csv). Some of those alerts may be coincident, because the scoring window accepts any alert raised from 28 days before to 90 days after the step.
5. Performance monitoring on matured labels lags input monitoring by 35 to 60 days where it fires at all. The CUSUM bias detector (h = 8, standardised on 2023 H1, budget 1 per horizon per month over 11 routes; 6 episodes in the no-break year 2023) first alerted on a break route at day 45 (IST-ADB, d = 7, 2024-04-15) and day 66 (IST-ADB, d = 30, 2024-05-06). IST-AYT and IST-FRA raised no new CUSUM alert at either horizon within 90 days (IST-FRA d = 30 was already in a CUSUM alert from 2024-02-24 to 2024-03-09, raised before the step). The d = 30 label for a departure matures on the departure date, 30 days after the forecast was issued, so by the decision date (day 45) the only alert known was IST-ADB d = 7. Matured error barely moved: break-route d = 30 WAPE was 0.0295 in February and 0.0331, 0.0295 and 0.0307 in March to May 2024, against 0.0268, 0.0259, 0.0253 and 0.0272 on other routes, and 0.0233 to 0.0344 across the months of 2023. Break-route bias, which sat between -0.9% and +0.3% in 2023, rose to +0.5%, +0.7%, +0.9% and +1.0% in September to December 2024, when the CUSUM streams for IST-AYT (2024-11-30) and IST-FRA (2024-12-19) finally fired at d = 30.
6. The retrain decision at day 45 is to keep v1 and watch, and later labels agree. On the 21 most recent departure days before 2024-04-15 (147 break-route and 252 other-route flight-days; models fitted on the 12 months to 2024-03-25, 6,935 rows), d = 30 WAPE on break routes was 0.0303 for v1, 0.0299 for retraining on the recent window, 0.0299 with a break indicator and 0.0994 for rolling back to additive pickup. The best alternative lowers break-route WAPE by 1.4%, against the 10% the rule demands, and changes other routes by +0.8%. At d = 7 the numbers are 0.0109, 0.0104, 0.0106 and 0.0307. Revisited on all departures after 2024-05-15 up to 2024-12-31 (1,610 break-route and 2,760 other-route flights at d = 30), v1 had WAPE 0.0299 on break routes and the break-indicator retrain 0.0300; on other routes 0.0252 against 0.0243. Retraining would not have helped the routes that the step hit, and the rollback to a simpler rule is about three times worse. The reason is structural: the model predicts final bookings as a ratio of the 30-day snapshot, which already carries the lower demand.
7. Alert accounting: 64 alerts, most of them performance or data-quality rather than drift. 29 input-drift alerts (13 act, 16 dismiss), 22 matured-label CUSUM alerts (all act by rule), 5 validation blocks (act) and 8 standing row-level data findings (7 act, 1 watch). Scored against the event log, 3 of the 29 drift episodes (10.3%) are on a stream the step can move inside the window, all three on competitor_capacity; the naive monitor has 7 of 124 (5.6%) and the tuned fixed-reference rule 6 of 155 (3.9%). The CUSUM has 2 of 22 (9.1%). Episodes per week over 44 streams: 1.19 naive, 0.29 designed drift, 0.21 CUSUM. The other 26 designed-drift episodes are not events in the log, which does not make them false: the simulator has a growth trend and calendar effects.
8. A 45-day label window dismissed two true shifts and "confirmed" coincidences. The competitor_capacity alerts on IST-AYT and IST-FRA were dismissed on 2024-04-21 and 2024-04-25 ("no matured-label alert within 45 days") and stayed open until 2024-12-31 (302 and 298 days firing) while the CUSUM for those routes fired late in 2024. In 2023, which has no event, 14 drift alerts were raised and 3 were marked act because a CUSUM alert on IST-ADB (2023-07-15, 88 days) was open nearby. A dismissal on short labels should be a watch with a revisit date, not a closure. The comp_cap alerts on the break routes also never close, because the matched reference is the pre-break year; accepting the shift needs a re-baseline.
9. Robustness. Reference padding of 0, 7 and 14 days gave 0.023, 0.034 and 0.045 episodes per stream-month in 2023 H2 and detected the step on competitor_capacity of 3 of 3 routes within 12, 10 and 8 days. Fixed-width instead of quantile bins, and 5 instead of 10 bins, did not change the 2023 H2 rate (0.034) or the delay (10 days). A simulated gradual rise of competitor prices (+1% per week from 2024-07-01): the naive monitor was already alerting on 11 of 11 routes and raised nothing new; the designed monitor raised on 11 of 11 routes after a median 32 days, at a median cumulative rise of 5.1%. The seasonal statistic was undefined on 1.4% of its stream-days (too few observations after blocked or missing days).

Key tables

Detection of the 2024-03-01 step (days after the step; first_delay/last_delay over the three break routes; "already open" counts streams whose alert was open before the detection window began):

Monitor	Feature	Detected of 3	Already open	First delay	Last delay
naive fixed 2022, as delivered	competitor capacity	0	3	-	-
naive fixed 2022, as delivered	booking curve d = 30	2	3	-8	64
naive fixed 2022, as delivered	forecast	3	3	25	64
seasonal matched, chosen rule	competitor capacity	3	0	6	10
seasonal matched, chosen rule	competitor price	1	0	53	53
seasonal matched, chosen rule	booking curve d = 30	0	0	-	-
seasonal matched, chosen rule	forecast	0	0	-	-
CUSUM on matured labels	d = 30	1	1	66	66
CUSUM on matured labels	d = 7	1	0	45	45

Alert volume in 2023 (episodes per route-feature stream-month; per feature counts one feature over its 11 routes):

Design	Window	Per stream-month	Per feature-month	Stream-days in open alert
naive fixed 2022	2023 full	0.152	1.672	0.965
fixed 2022, tuned rule (no cell within budget)	2023 H1 / H2 / full	0.200 / 0.110 / 0.154	2.198 / 1.206 / 1.693	0.147 / 0.197 / 0.172
seasonal matched, chosen rule (from 2023-02-03)	2023 H1 / H2 / full	0.0235 / 0.0340 / 0.0293	0.259 / 0.374 / 0.322	0.008 / 0.020 / 0.015
CUSUM on matured labels, h = 8 (22 streams)	2023 H1 / H2 / full	0.0154 / 0.0302 / 0.0228	0.169 / 0.333 / 0.251	0.005 / 0.035 / 0.020

Threshold trade-off on 2023 H1, seasonal-matched design (selected cell in bold; adjusted PSI threshold, KS p threshold; the last two columns are reported only and were not used for the selection):

PSI adj.	KS p	Per feature-month (H1)	Open share (H1)	Within budget	Max competitor-capacity delay (days)	Break streams detected of 9
0.05	0.05	2.537	0.126	no	6	9
0.05	0.01	1.553	0.048	no	8	9
0.05	0.001	0.259	0.008	yes	10	3
0.30	0.05	1.605	0.062	no	7	7
0.30	0.01	1.139	0.025	no	8	6
0.30	0.001	0.259	0.006	yes	10	3

Retrain options at the decision date 2024-04-15, WAPE on departures 2024-03-26 to 2024-04-15:

Option	d = 30 break routes	d = 30 other routes	d = 7 break routes	d = 7 other routes
v1 (keep)	0.0303	0.0257	0.0109	0.0093
retrain, recent 12 months	0.0299	0.0258	0.0104	0.0091
retrain, recent 12 months + break indicator	0.0299	0.0259	0.0106	0.0091
rollback, additive pickup	0.0994	0.0796	0.0307	0.0245

Decision record (outputs/decision_record.md): keep v1, status watch, owner "RM forecasting owner" (a simulated role), revisit 2024-06-30 or earlier on a CUSUM alert on IST-AYT or IST-FRA or a rolling WAPE above 1.25 times its 2023 level. The rule (at least 10% lower break-route d = 30 WAPE and at most 5% worse elsewhere) was not met.

Implementation notes that change how the numbers read

• The KS p-value is kstwo.sf(D, round(n m / (n + m))), which is what scipy.stats.ks_2samp(method="asymp") returns in scipy 1.18.1. An earlier version in the package used the limiting Kolmogorov distribution and disagreed with scipy (0.744 against 0.699 in the first failing test case); it was replaced and a test now checks equality to 1e-9.
• Rates are divided by the days on which a monitor is defined. The seasonal design is defined from 2023-02-03, so its "2023 H1" is 148 days, not 181.
• The first budget used 1 episode per route-feature stream-month. It met the criterion per stream but gave 1.45 episodes per feature-month, so the budget was tightened to the strict reading before the recorded run. 2023 H2 had been seen by then, so it is held out for the grid-cell choice only in the sense that the cell is selected on H1.
• The scoring window for the booking-curve and forecast streams starts 28 days before the step: the first break-affected d = 30 snapshot arrives 30 days before, and 3 consecutive firing days are needed. An alert raised earlier cannot be an effect of the step.
• CUSUM runs from 2023-01-01 because v1's 2022 residuals are in-sample.
• An input alert counts as confirmed when a matured-label alert on the same route is open at any time in the 45 days after it is raised; an already-open alert therefore counts as confirmation.

What this does not prove

• It does not show that this pipeline would find a real break. The event log (one competitor-capacity step on three routes, three simulated feed faults) is known only because the data are synthetic. One step, one seed and one set of fault dates give one realisation of each delay; no distribution over seeds or dates was computed, so 6, 8 and 10 days carry no interval.
• The simulated faults are extreme (units times 100, a missing column, a stale extract). Subtle faults would pass: the fare and price ranges are 0.5 times the 2022 minimum to 2 times the maximum, and revenue is allowed to differ from passengers times fare by 5%.
• The thresholds are conventions tuned on one grid of 18 cells and a 148-day window. The budget reading, the 10% open-alert guard and the CUSUM standardisation are choices made here. Observations inside a 28-day window repeat the same 19 flights and are not independent, so KS p-values and PSI are used as scores with tuned thresholds, not as calibrated tests.
• The strict budget reduces drift alerts to 0.32 per feature-month and also removes the demand-side detections; the sensitive cells detect more streams at several times the alert volume. Which side to take depends on what an alert costs and what a missed shift costs, and neither is priced here.
• "Not in the event log" is not "false alarm". Precision against a one-event log is a lower bound for monitors that also see real, unlisted shifts such as the simulator's growth trend.
• The retrain comparison uses 21 departure days (147 and 252 flight-days) and reports no confidence interval, so 1.4% is within noise. The forecast is an easy problem (d = 30 WAPE about 3%) because every booking-curve snapshot in the simulator is a noisy share of final bookings; with real, slower-reacting curves an input shift would translate into error differently. "Retraining does not help" is a statement about this feed and these four options.
• The replay ignores the feedback loop: a retrained model would change RM actions and future data. Labels of blocked batches are dropped, not re-delivered later, and the quality of the fallback forecast on blocked days is not computed, so the count of corrupted batches scored (0 against 5) is a count of exposure, not a measured harm; the model does not use fares.
• Decisions are generated by rules and the decision date is derived from the event log (step + 45 days), not from the first alert; roles and owners are simulated. Performance alerts do not close as a human review would: the CUSUM on IST-ADB d = 30 stayed open for 131 days at the end of 2024.
• The seasonal reference for 2025 would contain the break itself; the replay stops at 2024-12-31, so re-baselining is described but not tested.

Reproduce

cd python && uv run python -m reference.r10_drift_monitoring_pipeline.run            # full, ~1 to 2 minutes
cd python && uv run python -m reference.r10_drift_monitoring_pipeline.run --smoke    # about 25 s, temp directory
cd python && uv run pytest tests/test_r10_drift_monitoring_pipeline.py