Skip to content
ML Academy

Week 4 · Saturday — build and investigate

Forecasting studio 4: business metric, error slices and one controlled change

Project thread forecasting · linked lessons day 16, day 17, day 18, day 19, day 20

Milestone W4-a: a business-relevant metric next to MAE, bias and WAPE, a ranked error-slice table, one controlled change with a before/after record, and a written keep-or-revert decision. All of it uses the 2024-H1 validation window only.

  • 90 minsession length
  • 3timed blocks
  • 4tasks
  • —status

Session plan90 min

  1. 15′Recap and plan
  2. 60′Build - metric, slices, controlled change
  3. 15′Error analysis and record

Before you start

Prepared before the session (under 2 minutes):

  • outputs/val_predictions_w03.csv, outputs/run_w03.json and src/w03_fit.py from Week 3;
  • brief v1, with its hypothetical costs;
  • the Week 3 error hypothesis.

The environment is unchanged: uv run --project python python ... from the repository root. The sample database is the same seed-42 version; rebuild it with cd python && uv run python -m capstone.airline_sim only if it is missing. The test period stays unloaded.

Blocks3

  1. Block 1: Recap and plan

    15 min

    Re-read Saturday's Week 3 error hypothesis and the hypothetical costs from brief v1. From memory, write how MAE, WAPE and bias differ, and one condition under which each misleads (Day 16).

    Plan in three lines:

    • the business metric you will compute;
    • the slices you will inspect;
    • the single change you expect to make, and the result that would make you revert it.

    Writing the revert condition before seeing results is the point.

  2. Block 2: Build - metric, slices, controlled change

    60 min
    1. Minutes 0-15 - metrics (task metric-choice). Use outputs/val_predictions_w03.csv. Nothing is refitted yet.
    2. Minutes 15-35 - error slices (task error-slices). Rank the slices by volume x error (Day 18) and choose one failing component.
    3. Minutes 35-60 - one controlled change (task controlled-change). Change exactly one thing, keep the split and seed fixed, and re-run.

  3. Block 3: Error analysis and record

    15 min

    Write the error report and decision record (task error-report). Log the change in an experiment log, either outputs/experiments.csv or a tracking tool. Record:

    • the config;
    • the code version;
    • before and after overall;
    • before and after on the target slice;
    • the keep or revert decision.

    Add the next unresolved question to decision-log.md.

Tasks4

Sign in to save your work.

  1. Task 1

    analysis

    On the validation predictions, compute MAE, mean error (bias) and WAPE for the pickup baseline and for the candidate.

    Then compute one business metric tied to the D-14 decision. Either option is fine:

    • an asymmetric seat cost, using your hypothetical c_over and c_under;
    • a decision cost. Apply the rule "close L if the forecast is at least capacity" to the forecast and to the actual bookings_final, then cost the wrong decisions of each type.

    State one condition under which your chosen metric misleads.

    The expected evidence for this task is shown after you save your own output.

  2. Task 2

    analysis

    Slice the validation errors by market, by route, by month and by holiday or event flag. Also slice by near-capacity departures (bookings_final at or above capacity) against the rest, and by before versus after 2024-03-01 for IST-ADB, IST-AYT and IST-FRA against the other routes.

    For each slice, report the row count, MAE, bias and the total absolute error (volume x error). Rank by total absolute error. Pick the top actionable slice and state a root-cause hypothesis.

    The expected evidence for this task is shown after you save your own output.

  3. Task 3

    code

    Make exactly one change aimed at the chosen slice. Examples:

    • add one contract-approved feature;
    • switch the target to pickup;
    • make the baseline pickup seasonal (route x weekday x month);
    • restrict the training window.

    Keep the split, seed and everything else fixed. Re-run, and report the before and after values overall and on the target slice, for both MAE and the business metric.

    The expected evidence for this task is shown after you save your own output.

  4. Task 4

    writing

    Write notes/error-report-w04.md in 150-250 words. Cover:

    • the metric choice and why;
    • the top slices;
    • the hypothesis;
    • the change and its result;
    • the decision;
    • one limitation.

    Add a 3-line decision record: context, decision, consequence.

    The expected evidence for this task is shown after you save your own output.

Rubric4 criteria

CriterionMeetsExceeds
Metric fitMAE, bias and WAPE are computed, together with one business metric tied to the D-14 decision. One misleading condition is stated.Shows whether the business metric changes which method looks better, and explains why in terms of where the errors fall relative to capacity.
Error-slice prioritizationThe slices are ranked by total error and a testable root-cause hypothesis is given.The analysis separates model error from data or censoring effects, and from the documented 2024-03-01 regime change, before blaming the model.
Controlled experimentExactly one change is made. The split and seed are fixed, before/after is reported, and the decision follows the pre-stated revert rule.The experiment log is complete enough to reproduce both runs. The note discusses whether the improvement, if any, exceeds run-to-run or window-to-window noise.
Honest reporting"No improvement" is reported as such, and the costs are labelled hypothetical.The report names what evidence would overturn the decision.

Required output

Saved in forecasting-project/:

  • the metrics table, including the business metric with its labelled hypothetical costs;
  • the ranked slice table;
  • the controlled-change code and an experiment-log entry, with before/after and keep/revert;
  • notes/error-report-w04.md, with the decision record;
  • the next question in decision-log.md.

This is the Week 4 Saturday part of Appendix C: an error report and one controlled change.

Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.