r14 results - weekly batch forecasting platform prototype (project-025)

Recorded run of cd python && uv run python -m reference.r14_weekly_batch_forecast_platform.run (full mode). Every number below is copied from the files in outputs/ written by that run (summary.json, panel_summary.csv, accuracy_by_level.csv, accuracy_by_slice.csv, accuracy_restated_weeks.csv, release_gate.csv, replay_from_ids.csv, replay_negative_controls.csv, chaos_suite.csv, scaling_test.csv, cost_time_budget.csv, training_jobs.csv, vintage_index.csv, run_manifests.jsonl, success_criteria.csv, report.md). The panel is synthetic (thinned from the academy airline simulator), not M5 and not a retailer; costs and the SLA window are hypothetical.

Item	Value
Run date	2026-10-12
Seeds	database 42; panel 20241014; LightGBM random_state=42, deterministic=True, n_jobs=1
Python / numpy / pandas / scipy / lightgbm	3.12.13 / 2.5.3 / 3.0.6 / 1.18.1 / 4.7.0
Machine	Apple M5, macOS arm64, CPU only, local files as the "object store"
Runtime of the recorded run	73.5 s (production loop 33.0 s; other full runs during development took 73 to 130 s on a shared laptop). --smoke: 29 s.
Panel	1,140 series, 19 flights, 11 routes, 3 markets, 1,096 days, 1,133,849 series-day cells, 39.0% zeros, median series mean 1.07 units per day, 4,981 ledger corrections, 116 new and 67 discontinued series
Simulated production	26 weekly runs, issue dates 2024-06-10 to 2024-12-02, 28-day horizon, champion = statistical model, challenger = global LightGBM retrained every 4 weeks (7 training jobs), release decision at 2024-10-14
Tests	uv run pytest tests/test_r14_weekly_batch_forecast_platform.py (see the end of this file)

Findings

1. A rerun of the whole backfill changes nothing. The run key is a hash of issue date, requested model version, data snapshot id, code version and contract version. Re-running all 26 weeks skipped 26 of 26 (skipped_idempotent), left the hash of every file under vintage/ unchanged and reproduced all 26 forecast hashes; the rerun took 0.74 s against 33.0 s for the first production loop. A fallback run is not treated as done: the chaos suite reruns the same week without the fault and gets a normal success (rows of chaos_suite.csv).

2. A process killed during publish leaves nothing committed and the resume reproduces the clean backfill. Killed while publishing the 2024-09-09 vintage after 13 committed weeks: the killed week had no committed partition and no CURRENT pointer, one staging directory was left behind and removed by the next run. The resume skipped 13 weeks and recomputed 13; all 26 forecast hashes and all data files (everything but the manifests, which carry timings) equal those of the clean backfill. The same kill is a row in the chaos suite.

3. Every injected fault ends in a published, coherent, non-negative forecast, and 12 of 12 cases behaved as designed.

   Case	Outcome	Series WAPE of the published forecast
   clean run	success	0.438
   2 transient scoring errors	success after 3 attempts, identical forecast	0.438
   3 transient errors (retries exhausted)	fallback (seasonal naive)	0.453
   persistent model job crash	fallback	0.453
   model artifact missing	fallback	0.453
   late feed (newest 2 days missing)	blocked by contract, fallback	0.456
   3% duplicated rows	quarantined, identical forecast	0.438
   2% negative unit rows	blocked by contract, fallback	0.454
   calendar missing for 10 horizon days	blocked by contract, fallback	0.453
   challenger crashes	production unaffected, shadow failed	0.438
   +500 virtual seconds in scoring	success, shadow shed, budget breach alerted	0.438
   killed while publishing	nothing committed, rerun succeeds	0.438

   The fallback costs 3.4% to 4.3% more series-level WAPE than the clean champion run in these cases. After the artifact case the rerun without the fault still falls back, because the artifact stays missing; restoring it is a registry action, not a rerun (restore time 15.2 s of which 15 s is virtual back-off).

4. Forecasts are replayable from three ids. 26 of 26 vintages rebuilt from (issue date, model version, data snapshot id) matched both the stored manifest hash and the hash of the values read back from the partition files (3 decimal places). The negative controls were detected: a wrong snapshot id is refused, a different model version gives a mismatch, and one value edited in a stored partition gives a mismatch.

5. The release gate blocked the buggy candidate and the honest challenger; the challenger's later record is mixed. On the four matured vintages before 2024-10-14 (2024-08-26 to 2024-09-16) RMSSE relative to the champion was +3.9% (total), +1.3% (market), +0.9% (route), -0.4% (flight) and -2.0% (series). The tolerance of +2% at any level is exceeded at the total level and the series gain of 2% is not reached (-1.96%), so it was not promoted. A challenger whose calendar was joined 3 days off was blocked as well: +9.1% total and +4.4% market. Scored on all 26 vintages afterwards (shadow files), the challenger is slightly better at the top (total RMSSE 0.757 against 0.774) and at the series level (0.807 against 0.821), about 2% each, so the 4-vintage gate decision was within what the later record would call noise. The gate thresholds are hypothetical.

6. The champion is only a little better than the fallback. Pooled over 26 matured vintages, WAPE (statistical / challenger / seasonal naive): total 0.039 / 0.039 / 0.041, market 0.062 / 0.062 / 0.066, route 0.116 / 0.117 / 0.123, flight 0.152 / 0.152 / 0.162, series 0.453 / 0.455 / 0.475. The champion's improvement over the fallback is 4.3% (total), 5.6%, 6.3%, 6.2% and 4.6% (series). Series level WAPE is high because 39% of cells are zero and half of the series sell about one unit a day: bottom-third volume series have WAPE 1.34 for the champion (1.34 for seasonal naive), top third 0.34 (0.36). New series (under 90 days of history) are at 0.489 (0.467 challenger, 0.490 naive) against 0.449 for established series. Bias: +2.0% for the champion and +1.4% for the challenger (overforecast), +0.4% for the fallback.

7. A planted restatement is flagged for review but does not change the forecast. On 2024-09-16 the contract's restatement check reported 467 changed cells and a 26.1% change of ZX105 over the last 28 days, set review_required and the alert review_required:restatement, and the run proceeded. The 2024-09-16 vintage had series-level WAPE 0.461 against 0.452 on the other weeks for the champion (one week; no inference). The check only flags; one week of evidence cannot say whether the 2% review threshold is right.

8. Cost and time are far inside the hypothetical window at this scale, and the extrapolation is only a trend. Median weekly run 0.19 s for 1,140 series (ingest 0.004 s, contract 0.009 s, score 0.004 s, shadow 0.04 s per run; the rest is writing partitions), training job 3.1 s on 310,642 rows. Run time over 285, 570, 1,140 and 2,280 series fits an exponent of 0.98. Linear extrapolation to 42,840 series gives 7.3 s per run and 118 s per training job, which is 0.05% of the subset budget (383 s, the 4-hour window scaled by series share) and a hypothetical 0.0005 USD per week at 0.05 USD per CPU hour.

Success criteria (record of project-025)

Criterion	Result	Detail
rebuilds any past forecast exactly from its version ids	met	26 of 26 replayed vintages identical; 3 of 3 negative controls detected
blocks a candidate worse than the champion on any level by more than the tolerance	met	bug candidate blocked; the honest candidate also failed the gate
produces a coherent fallback forecast when the model job fails	met	12 of 12 chaos cases as designed; all coherent and non-negative
subset run finishes within a measured runtime budget extrapolated to the full catalogue	met (weak)	0.19 s against 383 s; 7.3 s extrapolated; assumptions are listed in cost_time_budget.csv
design document with at least two ADRs	not met	not produced by this package; it is the learner's deliverable (this file records the two decisions the numbers support: the fallback is cheap and close in accuracy; the gate needs more than 4 vintages)

What this does not prove

• A prototype of behaviour, not of scale or infrastructure. Local files stand in for object storage, a scheduler, a model registry and a warehouse. Atomicity rests on rename and os.replace on one local file system; object stores and distributed schedulers have different failure modes (partial listings, concurrent writers, clock skew). There is no concurrency, no authentication and no real network.
• The extrapolation to 42,840 series is a trend over four sizes. Memory, I/O, a real item-store hierarchy and a much heavier model are not exercised; the 0.19 s per run is dominated by file writes of a small panel. Training time at scale is a linear guess.
• The panel is much easier than retail data in one way and different in another. 1,140 series are thinned from 19 flights, so series within a flight share a common signal, far more than item-store series. The accuracy numbers say nothing about real hierarchical retail forecasting and the intermittent-demand behaviour is only partly realistic.
• The release gate has hypothetical thresholds and four vintages. A different tolerance, window or metric would give a different decision; the challenger's later record is nearly even. The deliberately buggy candidate is one bug.
• Matured actuals are post-correction values. Series are scored against the final values after late corrections, and the RMSSE scale uses the final values of the preceding year, not the snapshot held on the issue date. This is an evaluation convenience; it can move the denominators slightly and cannot change any forecast.
• Chaos cases are injected by this package. They show that the designed checks trip on the faults the designer thought of. They do not estimate the frequency or the variety of real incidents; late data of one day, a renamed column or silent unit changes are not all included.
• Calendar facts are assumed known in advance (holidays and announced events), and the fallback needs only weekday. A real platform would also face a wrong, not only a missing, calendar.
• Timings vary between runs. All committed timing columns come from this one run on a shared laptop; forecast values and hashes are deterministic and reproduce.

Reproduce

cd python
uv run python -m reference.r14_weekly_batch_forecast_platform.run          # about 1.5 to 2 minutes, writes outputs/
uv run python -m reference.r14_weekly_batch_forecast_platform.run --smoke  # about 30 seconds, temp directory
uv run pytest tests/test_r14_weekly_batch_forecast_platform.py