Skip to content
ML Academy

Week 4 · Bonus — optional extra studio

Bonus studio 5: classical-ML error analysis and debugging on a classifier

Project thread forecasting · linked lessons day 13, day 15, day 16, day 17, day 18

Optional extension of Week 4: a classifier audit with baselines, a ranked error breakdown, a threshold chosen on a separate tuning window, a tested explanation and a debug log whose predictions are scored against what was found. Synthetic data and hypothetical costs only.

  • 90 minsession length
  • 3timed blocks
  • 5tasks
  • —status

Session plan90 min

  1. 15′Predict before you look
  2. 60′Audit the classifier
  3. 15′Record and decide

Before you start

Prepared before the session (under 3 minutes). From the repository root, save the script below as fill_model_b05.py. Do not run it until the second block, because it prints the confusion matrix that the first block asks you to predict. Run it with uv run --project python python fill_model_b05.py; it takes well under a minute. It writes outputs/bonus05_val.csv (2024-H1 validation departures with probability p) and outputs/bonus05_tune.csv (a tuning window, with p).

# fill_model_b05.py - "will this flight fill?" classifier at D-45 (synthetic airline data). Run from the repository root.
import os
import sqlite3
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier

con = sqlite3.connect("python/artifacts/airline_sample.sqlite")
df = pd.read_sql("""
    SELECT f.date, f.flight_no, f.route, f.market, f.capacity, f.weekday, f.holiday, f.event, f.bookings_final,
           b.cumulative_bookings AS bk45
    FROM flight_day f JOIN booking_curve b ON b.date = f.date AND b.flight_no = f.flight_no
    WHERE b.days_to_departure = 45 AND f.date <= '2024-06-30'""", con)
df["month"] = pd.to_datetime(df["date"]).dt.month
df["route_c"] = df["route"].map({r: i for i, r in enumerate(sorted(df["route"].unique()))})
df["filled"] = (df["bookings_final"] >= df["capacity"]).astype(int)       # label: the flight filled

train = df[df.date <= "2023-09-16"]
tune = df[(df.date >= "2023-09-17") & (df.date <= "2023-11-16")]            # threshold is chosen here, never on val
val = df[df.date >= "2024-01-01"].copy()                                    # 2024-H1: scored once

F = ["bk45", "capacity", "weekday", "holiday", "month", "route_c"]
model = HistGradientBoostingClassifier(random_state=0).fit(train[F], train["filled"])
val["p"] = model.predict_proba(val[F])[:, 1]
tune_p = model.predict_proba(tune[F])[:, 1]

pred = val["p"] >= 0.5
tp, fp = int((pred & (val.filled == 1)).sum()), int((pred & (val.filled == 0)).sum())
fn, tn = int((~pred & (val.filled == 1)).sum()), int((~pred & (val.filled == 0)).sum())
print(f"train {len(train)}  tune {len(tune)}  val {len(val)}  filled rate val {val.filled.mean():.3f}")
print(f"threshold 0.5: accuracy {(tp + tn) / len(val):.3f}  TP {tp}  FP {fp}  FN {fn}  TN {tn}")
os.makedirs("outputs", exist_ok=True)
val.to_csv("outputs/bonus05_val.csv", index=False)
tune.assign(p=tune_p).to_csv("outputs/bonus05_tune.csv", index=False)

Use these hypothetical costs for the threshold task: a false positive (the flight is flagged to fill, so a discount is closed, but it does not fill) costs 60 units; a false negative costs 120 units. The colleague is simulated. The explanation to test in test-claim is theirs: "domestic routes are harder to predict, so we need more domestic data or a domestic-only model."

Blocks3

  1. Block 1: Predict before you look

    15 min

    A "will this flight fill?" classifier was built at D-45 on the synthetic airline data. A simulated colleague reports "88% accuracy, the model works". You will audit that claim.

    Before running anything, write (task debug-log, first entry):

    • the accuracy of the dumbest useful baseline, from the class rate you expect, and the accuracy of a one-feature rule you would build;
    • which error type you expect to dominate at a 0.5 threshold, and why;
    • the two slices where you expect most errors;
    • one thing in the setup that could make the 88% misleading.

    The entries stay unedited. The point is to find out where your intuition about classifier errors is wrong.

  2. Block 2: Audit the classifier

    60 min
    1. Minutes 0-10 - baselines and confusion (task baselines-confusion). Run fill_model_b05.py (see setup), then compare the model with a majority baseline and a one-feature rule.
    2. Minutes 10-30 - error anatomy (task error-anatomy). Where are the errors, by slice and by distance from the boundary? Rank them by count, not by rate alone.
    3. Minutes 30-45 - threshold and calibration (task threshold-calibration). Check whether the probabilities can be trusted, then choose a threshold without touching the 2024 validation rows to select it.
    4. Minutes 45-60 - test the colleague's explanation (task test-claim). One claim, one test, one verdict.

  3. Block 3: Record and decide

    15 min

    Finish debug-log with the diagnosis and one controlled follow-up you would run, ordered by expected value. Compare each prediction from the first block with what you found, and mark which were wrong. Finish with the decision: which threshold would you ship, what is the evidence, and which number would you not report.

Tasks5

Sign in to save your work.

  1. Task 1

    analysis

    Report, on the 2024-H1 validation rows: the filled rate, the accuracy of always predicting "does not fill", the accuracy, recall and precision of a one-feature rule you build from the booking ratio bk45 / capacity with its cut-off chosen on the training rows only, and the model's confusion matrix at 0.5 with accuracy, recall and precision. Say in two sentences whether "88% accuracy" supports "the model works".

    The expected evidence for this task is shown after you save your own output.

  2. Task 2

    analysis

    At threshold 0.5, list the errors: how many are false negatives and false positives; their distribution by distance from the capacity boundary (bookings_final - capacity); and rankings by market, by route and by one other slice you choose. Rank the slices by number of errors, then by error rate, and say when the two rankings disagree. State what share of errors a better threshold or a better model could realistically remove.

    The expected evidence for this task is shown after you save your own output.

  3. Task 3

    analysis

    Build a reliability table with deciles of p on the validation rows and say whether the probabilities can be read as probabilities. Then choose a threshold for the hypothetical costs 60 and 120 without using the 2024 validation rows to select it, and report the validation cost at 0.5, at your threshold, and at the value that the cost ratio implies in theory. Say which of the three you would ship.

    The expected evidence for this task is shown after you save your own output.

  4. Task 4

    analysis

    Test the colleague's claim "domestic routes are harder to predict, so we need more domestic data or a domestic-only model". Define what "harder" would look like in the data, run one test that could contradict the claim, and write a verdict in three sentences with the sample sizes. Say which follow-up you would run before collecting any data.

    The expected evidence for this task is shown after you save your own output.

  5. Task 5

    writing

    Keep the debug log in three entries: the predictions before you looked, the findings with numbers, and the decision. In the last entry mark each of your predictions as right, wrong or unclear, then state the single controlled follow-up you would run and the result that would make you drop it.

    The expected evidence for this task is shown after you save your own output.

Rubric5 criteria

CriterionMeetsExceeds
Baselines firstThe accuracy claim is compared with a majority baseline and a one-feature rule, and the learner says what the 88% does and does not show.Chooses the rule's cut-off on training rows only and reports where the rule beats the model.
Error anatomyErrors are split by type, by distance from the boundary and by at least three slices, ranked by count and by rate.Separates errors that are reducible from errors a D-45 feature set cannot reach, and sizes both.
Threshold disciplineThe threshold is selected without the scored validation rows, from a cost ratio and a separate window, and three costs are compared.Notes that the tuning window is small and the cost curve flat, and says what extra evidence would settle it.
Testing an explanationThe colleague's claim is tested with a design that could contradict it, with sample sizes and a verdict.Checks whether the slice rate survives conditioning on another variable, and picks the cheapest follow-up before collecting data.
Honest logPredictions are scored and at least one is marked wrong or unclear. The log names a controlled follow-up.Lists the numbers that should not be reported, and why.

Required output

Saved in forecasting-project/ (or a scratch folder):

  • the baselines and confusion table, the ranked error tables and the reliability table;
  • the threshold decision with its three validation costs and the selection rule;
  • the claim test and verdict;
  • debug-log.md with predictions scored.

This is an optional extension of Week 4; it never changes core or studio completion counts.

Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.