Skip to content
ML Academy

Week 4 · Sunday — consolidate and defend

Forecasting studio 4: drift and pipeline incident, runbook and end-to-end defense (Day 20 extended assessment)

Project thread forecasting · linked lessons day 16, day 17, day 18, day 19, day 20

Milestone W4-b: an evidence-based response to a staged drift and pipeline incident, a short runbook and decision record, and a defense of the current model choice. Together with Saturday, this is the Week 4 main evidence in Appendix C: an error report, one controlled change, and a short runbook and decision record.

  • 90 minsession length
  • 4timed blocks
  • 5tasks
  • —status

Session plan90 min

  1. 20′Spaced recall (assessment section 1)
  2. 40′Improve - staged incident (assessment section 2)
  3. 20′Decision defense (assessment section 3)
  4. 10′Weekly review and planning (assessment section 4)

Before you start

Prepared before the session (under 3 minutes):

  • outputs/val_predictions_w03.csv, or its Week 4 successor if the controlled change was kept;
  • the fitted pipeline from src/w03_fit.py, so it can score a modified input;
  • the Week 2 variant database, forecasting-project/data/w02_variant.sqlite. If it is missing, rebuild it with the block in studio-w02-sun's setup;
  • the seed-42 sample database.

The messages and people in the incident are simulated. The reference facts in stage 3 are documented behaviour of the synthetic simulator and of the Week 2 variant.

Blocks4

  1. Block 1: Spaced recall (assessment section 1)

    20 min

    This session is also the Day 20 extended assessment, in four saved sections. Submit each one before reading any reference facts or answer keys.

    Section 1: the cumulative prompts in task recall-cumulative, closed-book (12 minutes). Then check them against lessons 11-19 and the Week 1-2 notes (8 minutes), and log each miss with a correction.

  2. Block 2: Improve - staged incident (assessment section 2)

    40 min

    The incident is simulated and released in three stages. Do not skip ahead.

    1. Stage 1 (5 min) - commit (task incident-commit). Read the two simulated messages. Write your hypotheses, your first three checks and your immediate operational step. Do this before running anything.
    2. Stage 2 (20 min) - investigate (task incident-investigate). Run the checks against your validation predictions and data.
    3. Stage 3 (15 min) - reference facts and runbook. Read the reference facts in task incident-runbook. Classify each issue, choose the response, and write the runbook.

    If studio-w04-sat's controlled change was carried forward, do it in the first 20 minutes. Stage 2 then shrinks to the two checks that touch your slice.

  3. Block 3: Decision defense (assessment section 3)

    20 min

    Defend one end-to-end modelling decision using your baseline and error evidence (task defend-decision). This matches the Day 20 outcome. The panel is simulated. Answer each question in at most 80 words.

  4. Block 4: Weekly review and planning (assessment section 4)

    10 min

    Week 4 review in decision-log.md. Record:

    • lessons 16-20 status;
    • the Day 20 checkpoint result;
    • studio status;
    • whether the Week 2 routing trigger fired (if it did, record the adjustment within existing budgets);
    • next week's priority, which is to prepare the backtest.

Tasks5

Sign in to save your work.

  1. Task 1

    recall

    Answer closed-book:

    1. When is MAPE undefined or misleading? What would you use instead?
    2. What is the difference between permutation importance and an intervention effect?
    3. Why rank error slices by volume x error rather than by MAE?
    4. Data drift vs concept drift: one airline example of each.
    5. What is training-serving skew? Give one way it arises in a snapshot feed.
    6. What can you monitor when labels arrive 14 days late?
    7. What does PSI compare? What does it not tell you?
    8. Give one Week 1-2 item: why must the D-14 snapshot come from the cutoff and not from the latest load?

    The expected evidence for this task is shown after you save your own output.

  2. Task 2

    writing

    Two simulated messages arrive on a Monday.

    • An RM analyst writes that forecasts for IST-FRA "feel off since spring 2024".
    • Data engineering writes that the booking snapshot feed was re-delivered for departures 8-14 April 2024, and some D-14 rows may have been loaded twice.

    Before running anything, write:

    • (a) at least three hypotheses, covering data drift, concept drift or a structural break, a pipeline defect or training-serving skew, and noise;
    • (b) the first three checks, in order;
    • (c) the immediate operational step for this week's forecasts: do nothing, flag, or fall back to the pickup baseline on the affected routes.

    The expected evidence for this task is shown after you save your own output.

  3. Task 3

    analysis

    Run these checks and save the outputs:

    1. Residual mean and MAE by route, before and after 2024-03-01, for IST-ADB, IST-AYT and IST-FRA against the other routes.
    2. The distribution shift of bk_d14 on those routes before and after (PSI or KS), against the other routes.
    3. competitor_capacity per route by month around March 2024.
    4. The April re-delivery, in the variant feed. Count the departures whose bk_d14 differs between a "latest load wins" rule and the point-in-time rule. Score both inputs with your fitted pipeline and report the change in predictions.

    The expected evidence for this task is shown after you save your own output.

  4. Task 4

    writing

    Reference facts. Read these only after saving stage 2.

    • The simulator documentation states that competitor capacity on IST-ADB, IST-AYT and IST-FRA steps up 25% on 2024-03-01. This is a structural break in the demand process.
    • The re-delivered rows in the variant feed carry a loaded_at three days after the snapshot date and higher values.

    Then:

    1. Classify each issue: concept drift or structural break, data drift, pipeline defect, or training-serving skew.
    2. Choose a response with evidence. Options include adding competitor_capacity (only if the contract shows it is known at the cutoff), retraining with a recent-weighted or post-break window, a route-level fallback to the pickup baseline, and enforcing point-in-time dedup.
    3. Write notes/runbook-w04.md, of at most one page, covering:
      • the trigger signals;
      • the owner (simulated roles);
      • the triage checklist;
      • hypothetical thresholds;
      • the actions and the rollback;
      • verification;
      • a message to the RM team of at most 100 words;
      • the post-incident follow-up.

    The expected evidence for this task is shown after you save your own output.

  5. Task 5

    defense

    Simulated panel questions. Answer each in at most 80 words, citing your numbers.

    1. "Should RM use your ML candidate or the pickup baseline next month?"
    2. "Which metric did you optimize, and why is it the right one for closing class L?"
    3. "What is your biggest error slice, and is it the model's fault?"
    4. "How would you know, within two weeks, if the model broke in production?"
    5. "What would make you change your recommendation?"

    The expected evidence for this task is shown after you save your own output.

Rubric4 criteria

CriterionMeetsExceeds
Incident reasoningHypotheses are committed before evidence. At least three checks are run. Each hypothesis is marked with evidence. The two issues are classified correctly.Quantifies the effect of the re-delivery on predictions, and separates the structural break from noise using a comparison against unaffected routes.
Proportionate responseThe response follows the evidence and includes a rollback or fallback and a verification step.Checks feature availability before adding `competitor_capacity`, and states the cost of each option, such as reduced history after a post-break retraining window.
Runbook usabilityThe runbook has triggers, an owner, a triage checklist, actions, rollback and verification, on one page. The RM message is plain and short.Someone else could run it without the author. The thresholds are explicit and labelled as hypothetical starting values to calibrate.
End-to-end defenseAll five answers cite the learner's own evidence and are internally consistent.The answers anticipate follow-ups, and state what Week 5 must show for the recommendation to stand.

Required output

Four saved assessment sections: the recall answers; the stage 1 commitment with stage 2 outputs; the runbook; and the defense answers. Also updated notes/recall-log.md and a Week 4 entry in decision-log.md.

This completes the Week 4 main evidence: an error report, one controlled change, and a short runbook and decision record. It also serves as the Day 20 extended assessment.

Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.