Forecasting studio 3: review flawed model code and defend the validation design

Prepare before the session (2 minutes). Save the following as forecasting-project/review/flawed_w03.py. It is scaffolding with deliberate flaws, so do not reuse it.

# flawed_w03.py - a colleague's "D-14 forecast" (contains deliberate flaws; do not reuse)
import sqlite3
import numpy as np
import pandas as pd
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

con = sqlite3.connect("python/artifacts/airline_sample.sqlite")
df = pd.read_sql("""
    SELECT f.date, f.flight_no, f.route, f.weekday, f.holiday, f.capacity, f.fare_level,
           f.bookings_final, b.cumulative_bookings AS bk_on_hand
    FROM flight_day f
    JOIN booking_curve b ON b.date = f.date AND b.flight_no = f.flight_no
    WHERE b.days_to_departure = 7
""", con)

df["route_mean"] = df.groupby("route")["bookings_final"].transform("mean")
df["fare_level"] = df["fare_level"].map({"L": 0, "M": 1, "H": 2})
X = df[["weekday", "holiday", "capacity", "fare_level", "bk_on_hand", "route_mean"]]
y = df["bookings_final"]

X = StandardScaler().fit_transform(X)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, shuffle=True, random_state=0)

best = None
for depth in [2, 3, 4, 5]:
    m = GradientBoostingRegressor(max_depth=depth, random_state=0).fit(X_tr, y_tr)
    mae = mean_absolute_error(y_te, m.predict(X_te))
    if best is None or mae < best[0]:
        best = (mae, depth)

baseline_mae = mean_absolute_error(y_te, np.full(len(y_te), y_te.mean()))
print(f"model MAE {best[0]:.1f} (depth {best[1]}) vs baseline MAE {baseline_mae:.1f}")

Run it from the repository root with uv run --project python python forecasting-project/review/flawed_w03.py. It takes a few seconds. You also need src/w03_fit.py, outputs/val_predictions_w03.csv and the boundary note from Saturday. The colleague is simulated.