Skip to content
ML Academy

Week 2 · Bonus — optional extra studio

Bonus studio 1: causal inference and experiment review - what the data can and cannot say about price

Project thread forecasting · linked lessons day 4, day 5, day 6

Optional bonus. It adds no milestone and does not change core or studio completion. If you complete it, store the four outputs in forecasting-project/notes/ so that Week 6 (what-if questions) and Week 12 (defence) can cite them.

  • 90 minsession length
  • 5timed blocks
  • 5tasks
  • —status

Session plan90 min

  1. 10′Commit predictions and load the data
  2. 25′Observational estimates - how far can controls go?
  3. 20′Natural experiment - difference-in-differences with a placebo
  4. 25′Review a flawed experiment brief
  5. 10′Decision memo

Before you start

Prepare before the session (about 3 minutes).

  • Python through the repository environment: uv run --project python python <script>. pandas, scipy and statsmodels are already in python/pyproject.toml.
  • The seed-42 sample database at python/artifacts/airline_sample.sqlite (rebuild with cd python && uv run python -m capstone.airline_sim if it is missing). All data are synthetic.
  • The evaluation boundary from decision-log.md: training departures up to 2023-12-17, validation 2024-01-01 to 2024-06-30, test from 2024-07-01. This studio uses no test-period row.

Loader (use it as given, then write your own analysis):

import sqlite3, numpy as np, pandas as pd
import statsmodels.formula.api as smf

con = sqlite3.connect("python/artifacts/airline_sample.sqlite")
fd = pd.read_sql("SELECT * FROM flight_day WHERE date <= '2024-06-30'", con, parse_dates=["date"])
assert fd["date"].max() <= pd.Timestamp("2024-06-30")

The airline's pricing rule, the capacity cap and the competitor step are documented in CONTENT_GUIDELINES.md section 5. Nothing else about how the data were generated is needed.

Blocks5

  1. Block 1: Commit predictions and load the data

    10 min

    Write your predictions before any model is fitted (task commit-predictions, 5 minutes). Then load the data with the snippet in the setup and confirm the guard: the largest date in every frame is 2024-06-30 or earlier (5 minutes).

  2. Block 2: Observational estimates - how far can controls go?

    25 min

    Work on task confounded-elasticity.

    1. Minutes 0-12 - four regressions (steps 1 and 2 of the task).
    2. Minutes 12-18 - the fare-level table (step 3).
    3. Minutes 18-25 - the bias paragraph (step 4).

    Do not open any answer key before saving your table and paragraph.

  3. Block 3: Natural experiment - difference-in-differences with a placebo

    20 min

    Work on task natural-experiment.

    1. Minutes 0-5 - find the treated routes and the date (step 1).
    2. Minutes 5-13 - the estimate and the placebo (steps 2 and 3).
    3. Minutes 13-20 - what you will report (step 4).

  4. Block 4: Review a flawed experiment brief

    25 min

    Work on task experiment-review.

    1. Minutes 0-5 - read the brief once and write your first-pass concerns without computing anything.
    2. Minutes 5-17 - recompute every number the brief reports, and check each claim the brief makes about the setup against the data you have (the calendar table is available).
    3. Minutes 17-25 - verdict and fix list.

  5. Block 5: Decision memo

    10 min

    Work on task decision-memo. Write it for a reader who has not seen your tables.

Tasks5

Sign in to save your work.

  1. Task 1

    analysis

    Before you fit anything, write down, in notes/bonus01-predictions.md:

    1. The sign you expect for the coefficient of ln(avg_fare) in a regression of ln(bookings_final) on it, with no controls, and the sign after adding route fixed effects and calendar controls.
    2. Whether you expect the difference-in-differences estimate for the competitor step to be negative, positive or about zero, and a rough size in percent.
    3. One reason you expect each of your first two guesses could be wrong.

    Give a number or an interval for each guess. "Probably negative" is not a prediction.

    The expected evidence for this task is shown after you save your own output.

  2. Task 2

    code

    Use departures from 2022-01-01 to 2023-12-17 only. Let y = ln(bookings_final) and x = ln(avg_fare).

    1. Fit four ordinary-least-squares models of y on x and report the coefficient on x with a heteroskedasticity- robust (HC1) standard error for each:

      • (a) no controls;
      • (b) route fixed effects;
      • (c) (b) plus weekday, month, holiday, seasonality and an indicator for an event day;
      • (d) (c) plus ln(competitor_price).

    2. Repeat all four on departures that are not at the booking cap, meaning bookings_final below floor(1.06 * capacity). Report how many rows you dropped and say in two sentences what filtering on the outcome does to the meaning of the coefficient.

    3. Within each route, compare the mean of bookings_final for fare_level H and L. State in two sentences what this table does and does not show.

    4. Write one paragraph: using the pricing rule documented in CONTENT_GUIDELINES.md section 5, say whether your model (d) is still biased, in which direction and why, and whether adding more observed controls could fix it. Name one source of variation in price that would identify the effect.

    The expected evidence for this task is shown after you save your own output.

  3. Task 3

    code

    The data notes document a step in competitor capacity on some routes. Treat it as a natural experiment on demand.

    1. Using competitor_capacity, find which routes show the step and on which date. Show the evidence in one table (mean competitor_capacity per route before and after the date you propose, with the percentage change).
    2. Estimate a difference-in-differences for y = ln(bookings_final) with pre = 2024-01-01 to 2024-02-29 and post = 2024-03-01 to 2024-06-30. Use OLS with treated, post and their interaction, and standard errors clustered by flight_no. Report the interaction coefficient, its standard error and 95% interval.
    3. Run a placebo: the same model and calendar windows one year earlier (2023-01-01 to 2023-02-28 against 2023-03-01 to 2023-06-30), with the same treated routes. Report the same three numbers.
    4. Decide what you will report: the step-2 estimate, an adjusted estimate, or "not identified". Justify the decision with parallel-trends reasoning, and give two reasons the estimate may differ from the effect on unconstrained demand.

    The expected evidence for this task is shown after you save your own output.

  4. Task 4

    review

    Review the following experiment brief. All numbers, the site and the people in it are simulated for this exercise. The only real information you can check against is the airline calendar in the calendar table.

    Brief E-17 (simulated). Change: a "seats left at this fare" banner on the fare-selection step of the booking page. Hypothesis: the banner raises booking conversion.

    Design. Primary metric: bookings per session. Two arms assigned 50/50 by browser session at the first page view. Two-sided test, alpha 0.05, powered to detect an absolute lift of 0.25 percentage points from a 4.00% baseline at 80% power. Planned run: 19 days at about 5,250 sessions per arm per day, 2024-06-12 to 2024-06-30. The treatment arm logs a session when the banner script has loaded; the control arm logs a session at first page view.

    Monitoring. The results dashboard was checked on days 3, 6 and 9. The test was stopped the first time the dashboard showed p < 0.05. Dashboard p-values: day 3, 0.38; day 6, 0.11; day 9, 0.020.

    Result (stopped after day 9, 2024-06-12 to 2024-06-20).

    ArmSessionsBookingsConversion
    Control48,0001,9204.00%
    Treatment47,1002,0264.30%

    Reported: +0.30 pp (+7.5% relative), p = 0.020. Secondary metrics: ancillary attach rate 11.00% against 11.44% (5,280 of 48,000 against 5,388 of 47,100; p = 0.03); average fare shown, page-load time and 24-hour cancellation rate showed no significant difference (p = 0.64, 0.12, 0.81). 6,100 visitor ids appear in both arms.

    Recommendation. Ship to 100% of traffic.

    1. Recompute the primary result: both conversions, the lift, a two-sided p-value from a two-proportion z-test, and a 95% interval for the difference.
    2. Test the 50/50 allocation against the observed session counts.
    3. Recompute the planned sample size per arm and say how much of it the test had when it was stopped.
    4. List every threat to validity you can find in the brief, each with the evidence for it and the direction it pushes the result.
    5. Give a verdict (ship, do not ship, re-run) and the three most valuable changes for a re-run.

    The expected evidence for this task is shown after you save your own output.

  5. Task 5

    writing

    A revenue-management director asks, "Should we cut IST-AYT fares by 5% this summer?" Write a memo of at most 200 words that a non-statistician can act on. It must contain:

    • a recommendation (do it, do not do it, or test first);
    • the one piece of evidence from today that most influenced you, and the one claim it cannot support;
    • the size of the revenue change the 5% cut implies at the two ends of the documented elasticity range, and what else the sign depends on;
    • the design of the test you would run, with its unit of randomization and its stopping rule;
    • the gap between your first predictions and what the data showed.

    The expected evidence for this task is shown after you save your own output.

Rubric4 criteria

CriterionMeetsExceeds
Honest causal claimsThe learner does not present a confounded regression coefficient as a price effect, names the confounding mechanism, and says what would identify the effect.The learner also separates confounding from censoring and selection, and says in which direction each moves the estimate.
Quasi-experimental designTreated units and the date come from the data, a placebo is run and read, and the reported estimate states its interval and one caveat.The estimate is adjusted for the placebo difference, and the memo says exactly what the placebo can and cannot rule out.
Experiment reviewThe reported statistics are recomputed correctly, the allocation is tested, and at least four threats are named with evidence and direction.The review explains which threats void the result and which only weaken it, and proposes a re-run that fixes the most valuable ones.
Decision usefulnessThe memo makes a recommendation, ties it to one specific result, and states the test design.The memo includes the revenue arithmetic across the plausible range and the conditions under which the recommendation would flip.

Required output

Saved in forecasting-project/notes/:

  • bonus01-predictions.md, dated before any fit;
  • the elasticity table with its paragraph;
  • the difference-in-differences table with the placebo and the reporting decision;
  • the experiment review with the recomputed numbers, the threat list and the verdict;
  • the decision memo.

This studio is optional. It does not change any completion count.

Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.