Skip to content
ML Academy

Week 3 · Saturday — build and investigate

Forecasting studio 3: pickup baseline and one ML candidate on a protected split

Project thread forecasting · linked lessons day 11, day 12, day 13, day 14, day 15

Milestone W3-a. A runnable pickup baseline and one ML candidate, trained on departures up to 2023-12-17 and scored on the 2024-01-01 to 2024-06-30 validation window. Saved validation predictions and a run record. The 2024-H2 test stays protected.

  • 90 minsession length
  • 3timed blocks
  • 5tasks
  • —status

Session plan90 min

  1. 15′Recap and plan
  2. 60′Build - baseline, candidate, predictions
  3. 15′Error analysis and record

Before you start

Prepared before the session (under 3 minutes):

  • sql/w02_model_d14.sql, reconciled, and the evaluation-boundary note from Week 2.
  • Python through the repository's environment. Run scripts from the repository root with uv run --project python python forecasting-project/src/<file>.py. pandas, scikit-learn, LightGBM and XGBoost are already listed in python/pyproject.toml, so nothing needs installing.
  • The seed-42 sample database. Rebuild it if missing with cd python && uv run python -m capstone.airline_sim.

Scaffolding is limited to the environment. The loading code, baseline, model and evaluation are learner-authored.

Blocks3

  1. Block 1: Recap and plan

    15 min

    Re-state the evaluation boundary from decision-log.md from memory, then check it: training departures up to 2023-12-17, validation 2024-01-01 to 2024-06-30, protected test 2024-07-01 to 2024-12-31.

    Apply the routing decision from Week 2. If it allocated part of this studio to a weak concept, take that time from this block or from step 4 of the build block, and write which.

    Write a two-line plan: which single ML candidate you will fit and why it is a sensible challenger to the pickup baseline (Days 11-12).

  2. Block 2: Build - baseline, candidate, predictions

    60 min
    1. Minutes 0-10 - load and guard (task load-and-guard).
    2. Minutes 10-25 - pickup baseline (task pickup-baseline).
    3. Minutes 25-50 - one ML candidate (task ml-candidate). Use at most three configurations. Record every one you try, because each look at validation is a comparison (Day 4, multiple comparisons).
    4. Minutes 50-60 - save validation predictions and a run record (task save-predictions).

    The test period is not loaded today. If you need a number you can only get from the test period, write the question down for Week 6 instead.

  3. Block 3: Error analysis and record

    15 min

    Do task error-first-look. Then record in decision-log.md:

    • validation MAE and bias for both methods, exactly as computed;
    • the configurations you tried;
    • one hypothesis about where the candidate fails;
    • the next unresolved question.

    A candidate that does not beat the baseline is a valid result. Record it plainly.

Tasks5

Sign in to save your work.

  1. Task 1

    code

    In src/w03_fit.py, load the modelling table for departures up to 2024-06-30 only.

    1. Split it into training (departures up to 2023-12-17) and validation (departures 2024-01-01 to 2024-06-30).
    2. Assert the guard: the maximum date in every loaded frame is 2024-06-30 or earlier.
    3. Print the row count and the minimum and maximum date per partition.

    The expected evidence for this task is shown after you save your own output.

  2. Task 2

    code

    Implement the pickup baseline from the brief. The forecast for each departure is:

    bk_d14 + the mean pickup (bookings_final - bk_d14) for the same route and weekday.

    Compute the mean pickup on training departures only. Report validation MAE (seats per departure) and mean error (bias, forecast minus actual).

    The expected evidence for this task is shown after you save your own output.

  3. Task 3

    code

    Fit one ML candidate on the same split. Options include ridge regression, or gradient boosting in its scikit-learn, LightGBM or XGBoost form. Use only contract-approved features, for example bk_d14, route, market, weekday, month, holiday and capacity.

    • Put the preprocessing in a Pipeline / ColumnTransformer, fitted on training rows.
    • Choose and justify the target: bookings_final directly, or the pickup (bookings_final - bk_d14) with bk_d14 added back.
    • Try at most three configurations, and report every one.

    The expected evidence for this task is shown after you save your own output.

  4. Task 4

    code

    Write outputs/val_predictions_w03.csv with the columns date, flight_no, route, market, capacity, bookings_final, bk_d14, yhat_pickup and yhat_model.

    Write a run record, outputs/run_w03.json, with:

    • the code version (a git commit or a file hash);
    • the database file hash;
    • the random seed;
    • library versions;
    • the selected configuration.

    The expected evidence for this task is shown after you save your own output.

  5. Task 5

    analysis

    List the 10 largest absolute errors for each method, with route, date, capacity and bk_d14. Write two or three sentences on any pattern. You might look at near-capacity departures, specific routes, holidays, or dates after 2024-03-01. Then state one testable hypothesis for Week 4.

    The expected evidence for this task is shown after you save your own output.

Rubric4 criteria

CriterionMeetsExceeds
Protected evaluationThe guard assertion exists. The test period is never loaded. Preprocessing and the pickup table are fitted on training rows only.The split, gap and guard are in a single reusable function that later weeks import, so no script can bypass them.
Baseline and candidateThe pickup baseline and one ML candidate both run end to end on the same split. MAE and bias are reported for each. Every configuration tried is listed.The target choice is argued, with a reason tied to the problem, and the note explains how the candidate could extrapolate badly. For example, trees cannot predict beyond the range of the training targets.
ReproducibilityThe predictions file and the run record exist. Re-running gives the same numbers.The run record includes data and code hashes, and a single command reproduces everything.
Honest interpretationResults are reported as computed, including "no improvement" if that is what happened. Patterns are stated as hypotheses.The note says why the selected configuration's validation score is optimistic, and what the Week 5 backtest must show before the candidate can be trusted.

Required output

Saved in forecasting-project/:

  • src/w03_fit.py, runnable from the repository root;
  • outputs/val_predictions_w03.csv;
  • outputs/run_w03.json;
  • validation MAE and bias for the baseline and for every candidate configuration;
  • a decision-log entry with one error hypothesis and the next question.

This is the Week 3 Saturday part of Appendix C: a runnable baseline and validation predictions, with the final test kept protected.

Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.