Skip to content
ML Academy

Week 5 · Saturday — build and investigate

Forecasting studio 5: seasonal baseline, rolling-origin backtest and cutoff checks

Project thread forecasting · linked lessons day 21, day 22, day 23, day 24, day 25

Milestone W5-a: a seasonal naive baseline, one point-in-time-safe lag or rolling feature, automated cutoff availability checks, and a 12-origin rolling-origin backtest. The backtest covers the pickup baseline at D-14 and D-30, the seasonal naive, and the ML candidate at D-14, all on development departures.

  • 90 minsession length
  • 3timed blocks
  • 4tasks
  • —status

Session plan90 min

  1. 15′Recap and plan
  2. 60′Build - baselines, features, backtest
  3. 15′Error analysis and record

Before you start

Prepared before the session (under 3 minutes):

  • src/w03_fit.py, including the kept Week 4 change if there was one, and the split/guard function;
  • sql/w02_model_d14.sql, extended with bk_d30 (the cumulative_bookings at days_to_departure = 30). If it is not extended yet, the extra join is a one-line change made in the recap block;
  • an empty configuration skeleton, configs/backtest_w05.yaml, with these keys: origins, window (expanding or sliding), gap_days, test_length, horizons, models, features, metrics, seed, data_version and code_version. The skeleton is scaffolding; the values are learner-authored.

The environment is unchanged: uv run --project python python ... from the repository root, with the seed-42 database. Expect the backtest to run in about a minute or less for 12 origins on this data size. If it is much slower, reduce the candidate's size rather than the number of origins.

Blocks3

  1. Block 1: Recap and plan

    15 min

    Without notes, draw the rolling-origin timeline for one origin T. Mark:

    • the refit date, T - 14;
    • the training departures, those before T - 14;
    • the test departures, the month starting at T;
    • each test departure's own D-14 cutoff.

    Check the drawing against lesson 25. Then write down which of your current features would break if the horizon became 30 days.

    Plan in two lines: the backtest configuration you will fill in, and the one lag or rolling feature you will add.

  2. Block 2: Build - baselines, features, backtest

    60 min
    1. Minutes 0-10 - seasonal naive baseline (task seasonal-baseline).
    2. Minutes 10-25 - one lag/rolling feature and automated cutoff checks (task availability-checks).
    3. Minutes 25-55 - rolling-origin backtest (task rolling-origin). Write the configuration file first, then run it.
    4. Minutes 55-60 - save the results table. Do not tune anything inside the backtest loop today.

    The backtest uses development departures only (up to 2024-06-30). The 2024-H2 test stays protected.

  3. Block 3: Error analysis and record

    15 min

    Do task backtest-record. Look at the per-origin results before the averages: which origins and which segments dominate the error?

    Record in decision-log.md:

    • the configuration file name and the code version;
    • three observations, each with a number;
    • the next unresolved question.

Tasks4

Sign in to save your work.

  1. Task 1

    code

    Implement a seasonal naive forecast: the bookings_final of the same flight number on the departure 364 days earlier, which is the same weekday.

    • Join on dates, not on row offsets.
    • State why the value is available at the D-14 cutoff.
    • State which departures have no seasonal value, and how the backtest handles them.

    The expected evidence for this task is shown after you save your own output.

  2. Task 2

    code

    Add one lag or rolling feature. An example is the mean bookings_final of the same flight over departures 15 to 42 days earlier. Write down the availability rule: a departure's final bookings are known after it flies, so a feature for departure d may only use departures dated d - 15 or earlier. State this assumption.

    Classify every exogenous input as future-known, known at the cutoff, or unclear, citing the contract. The inputs are weekday, month, holiday, capacity, event, competitor_capacity and competitor_price.

    Write an automated check. For at least 50 sampled departures, recompute each lag or rolling feature using only rows dated before that departure's D-14 cutoff, and assert that the result equals the pipeline value.

    The expected evidence for this task is shown after you save your own output.

  3. Task 3

    code

    Fill in configs/backtest_w05.yaml and implement the backtest:

    • monthly origins from 2023-07-01 to 2024-06-01, which gives 12 origins;
    • an expanding window;
    • per origin T, refit at T - 14, using training departures before T - 14; the test departures are the month starting at T;
    • for the D-30 pickup baseline, training departures before T - 30.

    Models: pickup baseline at D-14 and at D-30, seasonal naive, and your ML candidate at D-14 with the new feature.

    Store a long table with one row per origin, model, horizon and market. Its columns are row count, MAE, bias, WAPE, and skill against seasonal naive (the model's MAE divided by the seasonal naive MAE, Day 25).

    The expected evidence for this task is shown after you save your own output.

  4. Task 4

    writing

    In notes/backtest-w05.md, write three observations, each backed by a number from backtest_w05.csv:

    • one about horizons (D-30 against D-14 pickup);
    • one about a segment (market or route), including the three routes with the 2024-03-01 break;
    • one about stability across origins.

    Then write the question you most need answered before freezing the design in Week 6.

    The expected evidence for this task is shown after you save your own output.

Rubric4 criteria

CriterionMeetsExceeds
Cutoff correctnessAn automated check proves that every lag or rolling feature uses only departures before each D-14 cutoff. The exogenous inputs are classified with reasons.The check is part of the pipeline, so it runs on every build. It also covers the seasonal naive and the horizon-specific gaps.
Backtest design12 monthly origins, an expanding window, horizon-specific gaps, and refit per origin. The configuration file matches the code.The configuration is the single source the code reads. The design discusses expanding against sliding windows in light of the 2024-03-01 break.
Results reportingA long results table by origin, model, horizon and market. Spread is reported, not only means. Skill against seasonal naive is included.The observations connect error differences to mechanisms: more information closer to departure, seasonality, or the regime change. Each is labelled as a hypothesis or as evidence, as appropriate.
Protected testNo departure after 2024-06-30 is loaded, and the guard confirms it.The Week 6 freeze criteria are drafted from backtest evidence, before any test data are seen.

Required output

Saved in forecasting-project/:

  • configs/backtest_w05.yaml;
  • the backtest code;
  • outputs/backtest_w05.csv;
  • the availability-check code with its output (zero mismatches, or mismatches documented and fixed);
  • the exogenous classification table;
  • notes/backtest-w05.md, with three numeric observations and the next question.

This is the Week 5 Saturday part of Appendix C: the backtest configuration and results, and demonstrated cutoff correctness.

Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.