Skip to content
ML Academy

Week 4 · Bonus — optional extra studio

Bonus studio 4: from a model result to a one-page decision memo

Project thread forecasting · linked lessons day 16, day 17, day 18, day 19, day 20

Optional extension of Week 4: a one-page decision memo that turns a model comparison into a recommendation, with its evidence table, a sensitivity check, and answers to three reader questions. It uses synthetic data and hypothetical costs throughout, and it never claims more than the validation window supports.

  • 90 minsession length
  • 3timed blocks
  • 5tasks
  • —status

Session plan90 min

  1. 15′Frame the decision and the reader
  2. 60′Build the memo
  3. 15′Reader challenge and revision

Before you start

Prepared before the session (under 3 minutes). From the repository root, save this as memo_inputs_b04.py (outside the content folders is fine). Do not run it until the second block, because its output is what the framing block asks you to commit to first. Run it with uv run --project python python memo_inputs_b04.py; it takes well under a minute and writes outputs/bonus04_val.csv, one row per validation departure with f_base, f_model, bookings_final, capacity and full.

# memo_inputs_b04.py - the "model result" you will write a decision memo about (synthetic airline data)
import os
import sqlite3
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor

C_FALSE, C_MISS = 60, 120   # HYPOTHETICAL cost units per flight: closed L but it did not fill / kept L open and it filled
con = sqlite3.connect("python/artifacts/airline_sample.sqlite")
df = pd.read_sql("""
    SELECT f.date, f.flight_no, f.route, f.capacity, f.weekday, f.holiday, f.bookings_final,
           b.cumulative_bookings AS bk14
    FROM flight_day f JOIN booking_curve b ON b.date = f.date AND b.flight_no = f.flight_no
    WHERE b.days_to_departure = 14""", con)
df["month"] = pd.to_datetime(df["date"]).dt.month
tr = df[df.date <= "2023-12-17"].copy()                      # labels known by the first forecast origin (2023-12-18)
va = df[(df.date >= "2024-01-01") & (df.date <= "2024-06-30")].copy()

ratio = (tr.bookings_final / tr.bk14.clip(lower=1)).groupby([tr.route, tr.weekday]).median().rename("ratio")
va = va.join(ratio, on=["route", "weekday"])
va["f_base"] = va.bk14 * va.ratio                            # pickup baseline

codes = {r: i for i, r in enumerate(sorted(tr.route.unique()))}
X = ["bk14", "capacity", "weekday", "holiday", "month", "route_c"]
tr["route_c"], va["route_c"] = tr.route.map(codes), va.route.map(codes)
va["f_model"] = HistGradientBoostingRegressor(random_state=0).fit(tr[X], tr.bookings_final).predict(va[X])

va["full"] = va.bookings_final >= va.capacity               # the flight filled
for name in ["f_base", "f_model"]:
    err = va[name] - va.bookings_final
    close = va[name] >= va.capacity                          # rule: close L when forecast reaches capacity
    miss, false = (~close & va.full).sum(), (close & ~va.full).sum()
    print(f"{name:8s} MAE {err.abs().mean():.2f}  bias {err.mean():+.2f}  WAPE {err.abs().sum() / va.bookings_final.sum():.4f}"
          f"  missed {miss}  false {false}  cost {C_FALSE * false + C_MISS * miss}")
print("validation flights:", len(va), " filled:", int(va.full.sum()))
os.makedirs("outputs", exist_ok=True)
va.to_csv("outputs/bonus04_val.csv", index=False)

Both forecasts are for the same 3,458 validation departures (2024-01-01 to 2024-06-30). The model was fitted on departures up to 2023-12-17 only. The cost units are hypothetical; the director and her opinions are simulated. You need no project files from Weeks 1-3 to do this studio.

Blocks3

  1. Block 1: Frame the decision and the reader

    15 min

    Do this before you run anything. A simulated reader, the Revenue Management director, will decide whether the "close the lowest fare bucket (L) when the D-14 forecast reaches capacity" rule should use the new model forecast instead of the pickup baseline. She reads one page and has not seen your notebooks.

    Write five lines (task frame-decision):

    • the decision, in one sentence, with its owner;
    • what she must do after reading (the ask);
    • the three questions she will ask first;
    • the result that would make you recommend "keep the baseline";
    • one number she will not care about, even though you found it interesting.

    Writing the "keep the baseline" condition before seeing the table is the point of this block.

  2. Block 2: Build the memo

    60 min
    1. Minutes 0-10 - get the result. Run memo_inputs_b04.py (see setup) and read what it prints. Do not write any prose yet.
    2. Minutes 10-25 - evidence table (task evidence-table). Build the one table the memo will stand on. It uses only numbers you computed or read from the script output.
    3. Minutes 25-50 - write the memo (task decision-memo). One page, recommendation first, 250-350 words.
    4. Minutes 50-60 - stress the headline (task sensitivity-check). Test whether the recommendation survives a different cost ratio and sampling noise, then correct the memo if it does not.
  3. Block 3: Reader challenge and revision

    15 min

    The simulated director's three questions are below. They are role-play prompts, not a real person's views. Answer each in at most 40 words, using numbers from your table (task reader-challenge).

    1. "Is this improvement real, or could a different six months have gone the other way?"
    2. "What do I do differently on Monday morning?"
    3. "What is the worst thing that happens if I follow you and you are wrong?"

    Then change the memo's first two sentences if any answer showed they overstate the evidence. Save the original and the revised version, and note what changed.

Tasks5

Sign in to save your work.
  1. Task 1

    writing

    Write the five framing lines before you run the script: the decision and its owner, the ask, the director's three first questions, the result that would make you recommend keeping the baseline, and one number she will not care about.

    The expected evidence for this task is shown after you save your own output.

  2. Task 2

    analysis

    Build the single evidence table for the memo. For the baseline and the model it must let the director compare the forecast quality and the decision outcome on the same departures. Add at least one view of where the result is different, chosen by you and justified in one line.

    The expected evidence for this task is shown after you save your own output.

  3. Task 3

    writing

    Write notes/decision-memo-b04.md, 250-350 words, one page. Order: recommendation (with the ask), the evidence table, what the evidence does not show, the risk if you are wrong, and what you will check next. A reader who stops after two sentences must know what you recommend and how sure you are.

    The expected evidence for this task is shown after you save your own output.

  4. Task 4

    analysis

    Test whether your headline survives. Choose the checks yourself, but include (a) a change to the cost ratio and (b) one way to express sampling uncertainty on the decision-cost difference between the two methods. State what you resampled or varied and what the memo says after the check.

    The expected evidence for this task is shown after you save your own output.

  5. Task 5

    defense

    Answer the three simulated reader questions in at most 40 words each, then revise the memo's first two sentences if needed. Keep both versions.

    The expected evidence for this task is shown after you save your own output.

Rubric5 criteria

CriterionMeetsExceeds
Recommendation firstThe memo opens with a recommendation and an ask that the reader can act on. Someone reading only the first two sentences knows what to do and how confident the author is.The ask includes a decision rule or a date for revisiting it, and the memo names who owns the next step.
Evidence disciplineEvery number comes from the learner's table, with its units, base and window. Hypothetical costs and synthetic data are labelled.The memo keeps forecast-quality and decision-value claims separate and shows why they differ in size.
Calibrated uncertaintyThe memo states what the single validation window can and cannot show, and the sensitivity check was run and reported, whichever way it came out.The recommendation's strength is matched to the interval, and the memo says what additional evidence would move it.
Reader focusThe memo fits one page, avoids method jargon the reader does not need, and anticipates her first questions.It leaves out a correct but irrelevant finding on purpose and says that it did.
RevisionThe reader answers are short and evidenced, and any change to the memo is traceable to a specific answer.The learner concedes a point against their own recommendation and narrows the claim.

Required output

Saved in forecasting-project/ (or a scratch folder if you have not built it):

  • notes/decision-memo-b04.md (original and revised first two sentences);
  • the evidence table and the sensitivity-check result;
  • the framing lines, the three reader answers and a one-line note on what you changed.

This is an optional extension of Week 4. It adds communication evidence; it does not change the Week 4 completion state.

Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.