Skip to content
ML Academy

Week 6 · Saturday — build and investigate

Forecasting studio 6: freeze the design, evaluate the held-out period once, decide under uncertainty

Project thread forecasting · linked lessons day 26, day 27, day 28, day 29, day 30

Milestone W6-a: a frozen forecast design chosen by pre-registered criteria, and one clean evaluation on the protected 2024-H2 departures against both baselines. Added to that: an interval with checked coverage, the close-or-keep-L decision implications under labelled hypothetical costs, and a report skeleton with computed tables.

  • 90 minsession length
  • 3timed blocks
  • 5tasks
  • —status

Session plan90 min

  1. 15′Recap and freeze
  2. 60′Build - held-out evaluation, uncertainty, decision
  3. 15′Error analysis and record

Before you start

Prepared before the session (under 3 minutes):

  • configs/backtest_w05.yaml, outputs/backtest_w05.csv, the repaired backtest (if any), and the freeze criteria in decision-log.md;
  • the backtest residuals per origin and market, saved by the Week 5 code. If they were not saved, re-run the backtest before the session; it is development data only;
  • the hypothetical costs c_over and c_under from brief v1, or the ones revised in Week 4;
  • the seed-42 database, unchanged since Week 1.

This is the only studio in which the guard is lifted, and only for the single held-out run. Day 28's project lesson already discussed the forecast design. Today produces its final evidence.

Blocks3

  1. Block 1: Recap and freeze

    15 min

    Re-read the pre-registered freeze criteria from Week 5. Apply them to outputs/backtest_w05.csv, or to the repaired backtest if it superseded yours, and choose the design.

    Write configs/frozen_w06.yaml with:

    • features;
    • model and hyperparameters;
    • target;
    • training window and gap;
    • refit policy for the test half-year: a single fit at 2024-06-16, or monthly refits at each origin minus 14 days;
    • interval method;
    • decision rule;
    • seed, data version and code version.

    Commit the file. From this point, nothing in it changes because of test results (task freeze-design).

  2. Block 2: Build - held-out evaluation, uncertainty, decision

    60 min
    1. Minutes 0-25 - held-out evaluation (task heldout-eval). Lift the guard for the first time, run the frozen design exactly once on departures 2024-07-01 to 2024-12-31, and score the pickup baseline and the seasonal naive on the same rows.
    2. Minutes 25-40 - uncertainty (task uncertainty). Build intervals from backtest residuals, then check coverage and width on the test.
    3. Minutes 40-60 - decision implications (task decision-implications).

    If something breaks, fix only the bug, not the design. Log the fix with its reason. Any design change after seeing test numbers is recorded as such, and the test result is then labelled "no longer clean".

  3. Block 3: Error analysis and record

    15 min

    Do task limitations-and-skeleton: test-period slices, a limitations list, and the report skeleton with the tables filled. Then record in decision-log.md:

    • the single test run's timestamp and code version;
    • the headline numbers as computed;
    • the next unresolved question.

Tasks5

Sign in to save your work.

  1. Task 1

    writing

    Apply the Week 5 freeze criteria and write configs/frozen_w06.yaml.

    In decision-log.md, state:

    • which criterion selected the design;
    • the runner-up;
    • what the result would have been under the fallback rule;
    • why no modern forecasting family (Day 27) was added. Either the evidence did not justify the cost of testing one, or it did, in which case say how it would be tested later against the same baselines.

    The expected evidence for this task is shown after you save your own output.

  2. Task 2

    code

    Train exactly as frozen. Training departures end on 2024-06-16 for a single fit, or follow the frozen refit policy. Evaluate once on departures 2024-07-01 to 2024-12-31.

    Report MAE, bias, WAPE and the Week 4 business metric for:

    • the frozen design;
    • the pickup baseline;
    • the seasonal naive.

    Report them overall, by month and by market. Save outputs/test_w06.csv with predictions and outputs/test_run_w06.json with the run record.

    The expected evidence for this task is shown after you save your own output.

  3. Task 3

    analysis

    Build an 80% interval for each test forecast from the empirical 10% and 90% quantiles of backtest residuals, per market if each market has enough residuals. This is a split-conformal-style approach. Then:

    • report the empirical coverage and the mean width on the test, overall and by market;
    • state the exchangeability assumption behind the method, and two reasons it may fail here (seasonality, and the 2024-03-01 break);
    • pick one example departure and explain its interval to an RM analyst in two sentences.

    The expected evidence for this task is shown after you save your own output.

  4. Task 4

    analysis

    Translate the forecasts into the close-or-keep-L decision on the test departures:

    1. Compute the critical ratio q* = c_under / (c_under + c_over) from your hypothetical costs.
    2. Use the q* quantile of each forecast distribution as the planning number. Close L when it reaches capacity.
    3. Compare the decision cost with:
      • the same rule driven by the pickup baseline;
      • the cost of the decisions that the actual bookings_final would have implied.
    4. State how censoring (bookings capped by capacity plus overbooking) biases this comparison on near-capacity departures.

    The expected evidence for this task is shown after you save your own output.

  5. Task 5

    writing

    Slice the test errors by market and month, with the near-capacity flag, and compare them with the Week 4 and Week 5 slices.

    Write the limitations list. Include:

    • synthetic data from one simulator;
    • one dataset version;
    • hypothetical costs;
    • censored demand;
    • the structural break;
    • a single test half-year;
    • the forecast's inability to answer price, capacity or competitor what-ifs.

    Create notes/report-w06.md with headings and the tables filled. Leave the prose for Sunday.

    The expected evidence for this task is shown after you save your own output.

Rubric4 criteria

CriterionMeetsExceeds
Evaluation integrityThe design is frozen and committed before the single test run. Both baselines are scored on the same rows. The run record matches the configuration.Test results are interpreted against the spread across backtest origins. Any post-test bug fix is logged transparently, with its effect.
UncertaintyThe interval is built from backtest residuals, with coverage and width reported on the test and the exchangeability caveat stated.Coverage is analysed by market or by month. Undercoverage is linked to a mechanism, with a proposed remedy such as recalibrating the residual quantiles on post-break data.
Decision analysisThe critical-ratio rule is applied with labelled hypothetical costs, and decision costs are compared with the baseline. Censoring is acknowledged.Shows how sensitive the conclusion is to the cost ratio, and explains how censored demand biases the comparison and what data would remove that bias.
Honest limitationsThe limitations list covers the data, costs, censoring, the break and causal what-ifs.Each limitation is tied to a concrete consequence for the recommendation, and to evidence that would remove it.

Required output

Saved in forecasting-project/:

  • configs/frozen_w06.yaml, committed before the test run;
  • outputs/test_w06.csv and outputs/test_run_w06.json, the single run;
  • held-out results against both baselines, overall and by slice;
  • the interval coverage and width table;
  • the decision-cost table with its labelled hypothetical costs and the censoring note;
  • the limitations list;
  • notes/report-w06.md, a skeleton with computed tables;
  • a decision-log entry.

This is the Week 6 Saturday part of Appendix C: frozen design, held-out evaluation, an uncertainty example and the decision implications.

Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.