Week 4 · Bonus — optional extra studio
Bonus studio 4: from a model result to a one-page decision memo
Project thread forecasting · linked lessons day 16, day 17, day 18, day 19, day 20
Optional extension of Week 4: a one-page decision memo that turns a model comparison into a recommendation, with its evidence table, a sensitivity check, and answers to three reader questions. It uses synthetic data and hypothetical costs throughout, and it never claims more than the validation window supports.
- 90 minsession length
- 3timed blocks
- 5tasks
- —status
Session plan90 min
Before you start
Prepared before the session (under 3 minutes). From the repository root, save this as memo_inputs_b04.py (outside
the content folders is fine). Do not run it until the second block, because its output is what the framing block
asks you to commit to first. Run it with uv run --project python python memo_inputs_b04.py; it takes well under a
minute and writes outputs/bonus04_val.csv, one row per validation departure with f_base, f_model,
bookings_final, capacity and full.
# memo_inputs_b04.py - the "model result" you will write a decision memo about (synthetic airline data)
import os
import sqlite3
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
C_FALSE, C_MISS = 60, 120 # HYPOTHETICAL cost units per flight: closed L but it did not fill / kept L open and it filled
con = sqlite3.connect("python/artifacts/airline_sample.sqlite")
df = pd.read_sql("""
SELECT f.date, f.flight_no, f.route, f.capacity, f.weekday, f.holiday, f.bookings_final,
b.cumulative_bookings AS bk14
FROM flight_day f JOIN booking_curve b ON b.date = f.date AND b.flight_no = f.flight_no
WHERE b.days_to_departure = 14""", con)
df["month"] = pd.to_datetime(df["date"]).dt.month
tr = df[df.date <= "2023-12-17"].copy() # labels known by the first forecast origin (2023-12-18)
va = df[(df.date >= "2024-01-01") & (df.date <= "2024-06-30")].copy()
ratio = (tr.bookings_final / tr.bk14.clip(lower=1)).groupby([tr.route, tr.weekday]).median().rename("ratio")
va = va.join(ratio, on=["route", "weekday"])
va["f_base"] = va.bk14 * va.ratio # pickup baseline
codes = {r: i for i, r in enumerate(sorted(tr.route.unique()))}
X = ["bk14", "capacity", "weekday", "holiday", "month", "route_c"]
tr["route_c"], va["route_c"] = tr.route.map(codes), va.route.map(codes)
va["f_model"] = HistGradientBoostingRegressor(random_state=0).fit(tr[X], tr.bookings_final).predict(va[X])
va["full"] = va.bookings_final >= va.capacity # the flight filled
for name in ["f_base", "f_model"]:
err = va[name] - va.bookings_final
close = va[name] >= va.capacity # rule: close L when forecast reaches capacity
miss, false = (~close & va.full).sum(), (close & ~va.full).sum()
print(f"{name:8s} MAE {err.abs().mean():.2f} bias {err.mean():+.2f} WAPE {err.abs().sum() / va.bookings_final.sum():.4f}"
f" missed {miss} false {false} cost {C_FALSE * false + C_MISS * miss}")
print("validation flights:", len(va), " filled:", int(va.full.sum()))
os.makedirs("outputs", exist_ok=True)
va.to_csv("outputs/bonus04_val.csv", index=False)
Both forecasts are for the same 3,458 validation departures (2024-01-01 to 2024-06-30). The model was fitted on departures up to 2023-12-17 only. The cost units are hypothetical; the director and her opinions are simulated. You need no project files from Weeks 1-3 to do this studio.
Blocks3
Block 1: Frame the decision and the reader
15 minDo this before you run anything. A simulated reader, the Revenue Management director, will decide whether the "close the lowest fare bucket (L) when the D-14 forecast reaches capacity" rule should use the new model forecast instead of the pickup baseline. She reads one page and has not seen your notebooks.
Write five lines (task
frame-decision):- the decision, in one sentence, with its owner;
- what she must do after reading (the ask);
- the three questions she will ask first;
- the result that would make you recommend "keep the baseline";
- one number she will not care about, even though you found it interesting.
Writing the "keep the baseline" condition before seeing the table is the point of this block.
Block 2: Build the memo
60 min- Minutes 0-10 - get the result. Run
memo_inputs_b04.py(see setup) and read what it prints. Do not write any prose yet. - Minutes 10-25 - evidence table (task
evidence-table). Build the one table the memo will stand on. It uses only numbers you computed or read from the script output. - Minutes 25-50 - write the memo (task
decision-memo). One page, recommendation first, 250-350 words. - Minutes 50-60 - stress the headline (task
sensitivity-check). Test whether the recommendation survives a different cost ratio and sampling noise, then correct the memo if it does not.
- Minutes 0-10 - get the result. Run
Block 3: Reader challenge and revision
15 minThe simulated director's three questions are below. They are role-play prompts, not a real person's views. Answer each in at most 40 words, using numbers from your table (task
reader-challenge).- "Is this improvement real, or could a different six months have gone the other way?"
- "What do I do differently on Monday morning?"
- "What is the worst thing that happens if I follow you and you are wrong?"
Then change the memo's first two sentences if any answer showed they overstate the evidence. Save the original and the revised version, and note what changed.
Tasks5
Task 1
writingWrite the five framing lines before you run the script: the decision and its owner, the ask, the director's three first questions, the result that would make you recommend keeping the baseline, and one number she will not care about.
Task 2
analysisBuild the single evidence table for the memo. For the baseline and the model it must let the director compare the forecast quality and the decision outcome on the same departures. Add at least one view of where the result is different, chosen by you and justified in one line.
Task 3
writingWrite
notes/decision-memo-b04.md, 250-350 words, one page. Order: recommendation (with the ask), the evidence table, what the evidence does not show, the risk if you are wrong, and what you will check next. A reader who stops after two sentences must know what you recommend and how sure you are.Task 4
analysisTest whether your headline survives. Choose the checks yourself, but include (a) a change to the cost ratio and (b) one way to express sampling uncertainty on the decision-cost difference between the two methods. State what you resampled or varied and what the memo says after the check.
Task 5
defenseAnswer the three simulated reader questions in at most 40 words each, then revise the memo's first two sentences if needed. Keep both versions.
Rubric5 criteria
| Criterion | Meets | Exceeds |
|---|---|---|
| Recommendation first | The memo opens with a recommendation and an ask that the reader can act on. Someone reading only the first two sentences knows what to do and how confident the author is. | The ask includes a decision rule or a date for revisiting it, and the memo names who owns the next step. |
| Evidence discipline | Every number comes from the learner's table, with its units, base and window. Hypothetical costs and synthetic data are labelled. | The memo keeps forecast-quality and decision-value claims separate and shows why they differ in size. |
| Calibrated uncertainty | The memo states what the single validation window can and cannot show, and the sensitivity check was run and reported, whichever way it came out. | The recommendation's strength is matched to the interval, and the memo says what additional evidence would move it. |
| Reader focus | The memo fits one page, avoids method jargon the reader does not need, and anticipates her first questions. | It leaves out a correct but irrelevant finding on purpose and says that it did. |
| Revision | The reader answers are short and evidenced, and any change to the memo is traceable to a specific answer. | The learner concedes a point against their own recommendation and narrows the claim. |
Required output
Saved in forecasting-project/ (or a scratch folder if you have not built it):
notes/decision-memo-b04.md(original and revised first two sentences);- the evidence table and the sensitivity-check result;
- the framing lines, the three reader answers and a one-line note on what you changed.
This is an optional extension of Week 4. It adds communication evidence; it does not change the Week 4 completion state.
Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.