Core day 28 of 60 · Week 6 · Part I · evaluation depth

End-to-End Forecasting Project

Your Weeks 1-6 forecast has a held-out result, a band and a decision rule. The commercial director asks about IST-TZX: what if we fly 180 seats instead of 160, what if we cut fares 5%, what if the competitor adds 25% capacity? Your fitted model returns a number for each. Which can you defend, and what do you say about the rest?

By the end: You write a one-page defense of your forecasting project (decision, cutoff, bar, held-out result, band, limits) and a triage of three what-if questions that says, for each, whether the model's number can be used and what evidence would be needed otherwise.

At a glance

Defend the forecast with its evidence and limits, and triage every what-if by how the driver was set, whether it varied and whether the outcome is capped.

Forecast
Sales, claims or bookings given the world as recorded: same pricing rule, capacity, competitors.
What-if
A changed input; the model's number is causal only if history varied it freely.
Confounded price
Fares followed demand, so the fare term has the wrong sign (+0.50 vs -1.00).
Censored capacity
Sold-out flights and empty shelves record the cap; capacity that never varied shows no gain.
Defense card
Decision, cutoff, bar, held-out result, band, limits, and what it cannot answer.

Recall · 5 min

From Day 5: why does a regression of bookings on fare in history point the wrong way, and what design fixes it? From Day 2: what is the target of the D-14 project, and why is it not demand?

Open due reviews

Concept · 10 min

A forecast answers "what will bookings be, given the world works as it did when the data were recorded": the same pricing rule, seat count and competitors. A what-if changes one of those. A model returns a number for any value you type into a column; whether it means anything depends on how that column got its values in history, not on held-out accuracy.

The model gets the competitor what-if roughly right and the price and capacity what-ifs wrong, for different reasons

Predict first

For a 5% fare cut on IST-TZX, which direction does the forecasting model predict for passengers?

Text description of the figure

A precomputed grid for IST-TZX with controls for capacity (160, 170, 180), price change (-10% to +10%) and competitor capacity (+0% or +25%). For each setting a readout and a line chart over the price grid compare the forecasting model's predicted change in passengers and revenue with the simulator's true change.

Two questions that look alike Overview

Forecast
What will bookings be if the world keeps working as it did?
What-if
What would bookings be if we set one input differently, all else equal?
Confounding
A pricing rule reacted to demand, so fare carries demand.
Censoring
Passengers stop at the bookable seats; turned-away demand is invisible.

The what-if triage, one driver at a time Diagram

Set by a demand rule?
- yes: to Confounded
no
Varied in history?
- no: to No information
yes
Outcome capped?sold out, stockout
- yes: to Unconstrain first
no
Forecast, same policyusable, with its band
Confoundedneeds an experiment or a design (Day 5)
No informationthe model cannot see the new value
Unconstrain firstbooking-curve pace, sold-out history

Price fails the first question; capacity on IST-TZX fails the second and third; competitor capacity passes the first two and is only partly capped, so it stays usable as a forecast with its band.

Passengers per flight-day: model versus simulator truth (IST-TZX) Chart

Price -5%: modelwrong sign-3.43
Price -5%: truth4.01
Capacity 180: modelsees no gain0.04
Capacity 180: truth5.09
Competitor +25%: modelratio 1.14: roughly right-12.11
Competitor +25%: truth-10.64

Today's run, synthetic. Bar length shows size; the sign is in the value.

Why the fare term points the wrong way, computed Illustration

high unseen demand shock
low shock
true effect of a fare change (dashed)

Each dot is a flight-day. Busy days get higher fares and still sell more, so the fitted line slopes up (+0.60); the true response to a fare change, all else equal, slopes down (dashed, -1).

The fare coefficient from three regressions on the same rows. Adding the known drivers helps only partly; the sign flips only when the unseen shock is held fixed, which no real dataset allows. An experiment supplies that variation by design.

Synthetic: 600 flight-days; log fare = 0.55 x known drivers + 0.5 x unseen shock + small independent variation; log bookings respond with elasticity -1. Same rule weights as the scenario-lab simulator, not its data, so the numbers differ from the lesson table.

A pricing rule that follows demand makes busy, expensive days sell more; only variation the rule did not choose recovers the true slope.

On a capped flight you record the cap, not the demand Illustration

Latent demand per flight-day. Everything right of the cap is recorded as 169: 34% of days sell out, and 4.8 passengers per day on average are turned away without leaving a row in the data.

True expected sales rise with bookable seats (curve). History holds a single point, 169 seats, so a model fitted on it has no slope to learn: in its world more seats change nothing. Raising the cap to 190 adds 4.0 sales per day here.

Estimating the demand level from capped sales. Both naive means are biased low; treating sold-out days as "at least 169" in a censored-normal model (EM) lands close to the truth, if demand really is roughly normal.

Synthetic: 2,000 flight-days with normal demand (mean 160, sd 22) and 169 bookable seats (seeded). Not the scenario-lab simulator; the cap matches it. EM assumes the normal shape; real unconstraining also uses booking-curve pace.

What is turned away, why a model of capped sales sees no value in more seats, and an estimate that treats sold-out days as lower bounds.

The one-page defense card, in this order Sequence

Decision and unit
Close fare class L at D-14, per departure; target bookings_final.
Contract
The D-14 cutoff snapshot and the features available then.
The bar
Pickup baseline and seasonal naive on rolling origins with a gap; one held-out run scored once.
Uncertainty
Band from backtest residuals, held-out coverage and width, planning quantile and its hypothetical costs.
Limits
Censoring, the 2024-03-01 break, one simulator, hypothetical costs, one test half-year.
What it cannot answer
Price and capacity what-ifs, with the evidence that would answer them.

Each line points to a saved file from the Week 6 studio.

Defense card: decision, contract at the cutoff, the bar, uncertainty, limits, and what the forecast cannot answer.
What-if triage, per driver: how was it set in history, did it vary, is the outcome observed or capped?
Two traps under every airline what-if: censoring (full flights show the cap, not demand) and a pricing rule that makes demand a Confounder (Day 5).
Competitor capacity moved outside the airline's own rule, so history holds usable variation; even then the answer is a forecast under unchanged policy.

Assumptions

The simulator's pricing rule, elasticities and cap are known to us (synthetic); in a real airline they are not, and every claim below would need an experiment or a quasi-experiment.
The scenario model is a pooled log-linear regression over all routes; a richer forecasting model changes the numbers but not the triage.
The "truth" is the simulator's response to the same random draws with only the input changed, for IST-TZX January-June 2025.

Application and workplace decision · 10 min

Predict first: for which of the director's three questions will the model's number have the wrong sign? Then run the code.

It holds the scenario-lab grid for IST-TZX: a pooled forecasting model fitted on 2022-2024, scored on January-June 2025 (synthetic).
It prints the model's predicted change next to the simulator's true change per flight-day, their ratio and the half-year revenue gap.
Finish by writing the one-sentence answer you would give the director for each question.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import math

# Copied from public/data/scenario-lab.json (python/precompute/scenario-lab.py; synthetic IST-TZX, 2025-01..06).
# Change per flight-day versus the base case: the fitted forecasting model vs the simulator's true response.
base_rev, flight_days = 13399.4, 362
what_ifs = {  # name: (model passengers, true passengers, model revenue EUR, true revenue EUR)
    "price -5%": (-3.431, 4.014, -1003.8, -300.8),
    "capacity 160 -> 180": (0.039, 5.086, 5.4, 537.5),
    "competitor capacity +25%": (-12.107, -10.638, -1209.2, -1035.4),
    "capacity 180 and competitor +25%": (-12.107, -8.561, -1209.2, -808.3),
}
print(f"{'what-if':32s} {'model':>7s} {'truth':>7s} {'model/truth':>11s}  {'model rev':>9s} {'true rev':>8s}  true-minus-model, half-year")
for name, (mp, tp, mr, tr) in what_ifs.items():
    print(f"{name:32s} {mp:+7.2f} {tp:+7.2f} {mp / tp:+11.2f}  {mr:+9.0f} {tr:+8.0f}  EUR {(tr - mr) * flight_days / 1000:+,.0f}k")

print(f"\nfare term: model {+0.504:+.2f} vs truth {-1.0:+.2f}")
print(f"competitor-capacity term: model {-0.42:+.2f} vs truth {-0.12 / math.log(1.25):+.2f} per log unit")
print(f"true capacity gain: {537.5 / base_rev:.1%} of base revenue per flight-day, EUR {537.5 * flight_days / 1000:,.0f}k per half-year")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 32 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

Predictive versus do() estimands, definedadvanced4 minThe two questions a model can be asked, written as math, and the one arrow in the causal graph that tells them apart.

Two estimands that share a regression line Formula

\begin{aligned} \tfrac{\partial}{\partial f}\,\mathbb E[Y \mid F = f] &= \beta + \gamma\,\delta \\ \tfrac{\partial}{\partial f}\,\mathbb E[Y \mid do(F = f)] &= \beta \end{aligned}

$\mathbb E[Y \mid F = f]$: predictive: what sales look like on days when the rule chose price f; right for a forecast under the same rule
$do(F = f)$: interventional: set the price yourself, cutting the arrow from demand to price; right for a what-if
$\delta$: Cov(f, d) / Var(f): how strongly the rule ties price to demand

Linear model y = beta f + gamma d + noise. The two agree only when gamma delta = 0: nothing that drives Y also chose X. Held-out accuracy certifies the left one; a what-if asks for the right one. Same split for a credit limit set by a risk rule, a discount sent by a targeting model, a treatment chosen by a clinician.

Observing the fare versus setting it: do() cuts one arrow Diagram

Price testyou set the fare: do(F = f)
sets it
Farein history: set by the rule
- effect -1: to Bookings
Expected demandnot a column in the data
- cut by do(): back to Fare
more demand
Bookingsthe forecast target

In history the dashed arrow is live: high fares coincide with strong demand, so a model of bookings on fare learns a positive slope. Setting the fare from outside removes that arrow, and fare no longer carries information about demand; only then does the slope answer a what-if.

The omitted-variable bias, derived and workedadvanced7 minThe bias derived in four lines, a four-day pricing example by hand, its direction by sector, why more data does not help, and the fixes.

The omitted-variable bias, derived in four lines Sequence

The true model
y = beta f + gamma d + e, with e uncorrelated with f and d. The analyst regresses y on f alone.
The short slope
beta-hat = Cov(f, y) / Var(f), the least-squares slope of a one-variable regression.
Substitute
Cov(f, beta f + gamma d + e) / Var(f) = beta + gamma Cov(f, d) / Var(f) + 0.
Read the bias
gamma x delta, where delta is the slope of d on f. Its sign is sign(gamma) x sign(delta); it is zero only if one of them is.

Adding d to the regression removes the term, but only if d is measured; the unseen demand shock never is. A randomized price makes delta = 0 by design.

Worked by hand: four days of an online shop's dynamic pricing (illustrative, log units) Table

Day	Demand signal d	Rule noise u	Log price f = d + 0.5u	Log sales y = -f + 2d
1	-1	+1	-0.5	-1.5
2	-1	-1	-1.5	-0.5
3	+1	+1	+1.5	+0.5
4	+1	-1	+0.5	+1.5
Short regression	sum fy = 3.0, sum f^2 = 5.0	-	slope +0.6	true beta = -1: wrong sign
Bias by the formula	sum fd = 4.0, delta = 0.8	gamma = 2	-1 + 2 x 0.8 = +0.6	matches the short slope
Use only u (variation the rule did not choose)	sum uy = -2, sum uf = 2	-	-2 / 2 = -1.0	recovers beta

The shop raises prices on high-demand days. A regression of sales on price says higher prices sell more; regressing on f and d together, or using only the random part u, returns -1. All slopes recomputed in Python.

The short slope is the true effect plus one bias term Bridge

True effect beta-1 log units
Omitted demand: gamma x delta2 x 0.8: the rule raises price on busy days+1.6 log units
Short-regression slope0.6 log units

The four-day shop above: beta = -1, gamma = 2, delta = 0.8, so the regression of sales on price alone returns -1 + 1.6 = +0.6. Randomizing the price sets delta = 0 and removes the bar in the middle.

Which way the bias points, by sector Map

Understated (marketing)

Discounts go to low-intent customers, who buy less: the discount looks weaker than it is.

Overstated (retail pricing)

Prices follow demand, which lifts sales: a price cut looks harmful.

Overstated harm (healthcare)

A programme goes to sicker patients (low health d): it looks like it causes readmissions.

Overstated benefit (SaaS)

Onboarding calls go to engaged accounts, who churn less: calls look to cut churn more than they do.

Sign of the bias = sign(gamma) x sign(delta). Before reading any coefficient as a lever, name d and guess both signs.

More data buys a precise wrong answer: the short slope settles at +0.6, not -1 Illustration

Middle 95% of the short-regression slope across 600 simulated samples per size. At 50 days it spans 0.25 to 0.92; at 3,000 days 0.56 to 0.64. The band shrinks onto +0.6, a precise wrong answer: the true effect, -1, never enters it.

Simulation of the worked table's process with standard normal d, u and noise e (seed 2828, 600 samples per size). Bias = gamma x delta = 2 x 0.8 = 1.6, whatever the sample size.

The worked table's process with normal draws, simulated from 50 to 3,000 days. The band narrows around beta + gamma delta = -1 + 2 x 0.8 = +0.6 and never moves towards -1: variance shrinks, bias stays.

The bias is a product: each fix removes one factor Hierarchy

Bias: gamma x deltafirst ask who set the input
- make delta = 0Randomize the inputa price test: no link to d
- measure dAdjust for donly the measured part; the shock stays
- use only uExogenous variationan instrument: tests, outages, rollouts
- more dataNothing changesvariance shrinks, the bias stays

A large R-squared of the input on known demand drivers means a rule follows demand. The u column of the worked table is the instrument idea: the part of price the rule did not choose recovers beta = -1.

Censored demand, formalized and unconstrainedadvanced6 minWhat a capped observation is, its likelihood, the conditional expectation that unconstrains it, six days worked through EM by hand, and the evidence ladder.

Unconstraining a sold-out day Formula

\mathbb E[D \mid D \ge c] = \mu + \sigma\,\frac{\varphi(a)}{1-\Phi(a)}, \qquad a = \frac{c-\mu}{\sigma}

$Y = \min(D, c)$: what is recorded, with a flag for D >= c: a sold-out day is a lower bound, not a value
$c$: the cap: bookable seats, shelf units, call-queue capacity, a policy limit on a claim
$\mu, \sigma$: mean and spread of demand, re-estimated in each EM round
$\varphi, \Phi$: normal density and distribution function
$\text{worked}$: mu = 150, sigma = 20, c = 169 (seats or shelf units): a = 0.95, phi/(1 - Phi) = 0.254/0.171 = 1.49, so E = 150 + 20 x 1.49 = 179.7

A sold-out day at 169 stands for about 180 on average (computed in Python). EM replaces each sold-out day by this expectation and refits until the estimates settle. It is one of the unconstraining methods compared by Weatherford and Polt (2002); it relies on the shape you assume.

The likelihood of capped data, and what EM maximizes Formula

L(\mu, \sigma) = \prod_{\text{open days}} f(y_i) \prod_{\text{capped days}} \bigl[1 - F(c)\bigr]

$f(y_i)$: density of the recorded value on a day that did not sell out: the value is exact
$1 - F(c)$: probability that demand reached the cap: all a sold-out day tells you

The Tobit form. EM maximizes it by filling each capped day with E[D | D >= c] and refitting; dropping the capped days instead deletes exactly the high-demand days (154.0 in the table below, against 159.0 naive).

Worked by hand: six days of a retail SKU whose shelf holds 169 units (illustrative) Table

Estimate	Rule	Days 141 152 160 169* 169* 163	Mean demand
Naive mean	treat 169* as the demand	(141 + 152 + 160 + 169 + 169 + 163) / 6	159.0
Drop sold-out days	treat them as missing	(141 + 152 + 160 + 163) / 4	154.0: the busiest days are gone
EM round 1	169* -> E[D \| D >= 169] at mu = 150, sigma = 20	169* -> 179.7	(616 + 2 x 179.7) / 6 = 162.6
EM round 2	recompute the expectation at mu = 162.6	169* -> 182.8	163.6
EM converged	repeat until mu stops moving (11 rounds)	169* -> 183.2	163.7

* marks a sold-out day. Sigma is held at 20 to keep the arithmetic on paper; full EM also updates sigma from the completed squares. Under a normal model each sold-out day hides about 14 units the shelf never recorded, and dropping those days biases the mean the other way. The same arithmetic unconstrains sold-out flights. Recomputed in Python.

Evidence for the capacity question, cheapest first Sequence

Unconstrain historical demand
From booking-curve pace and sold-out departures: a flight full at D-20 hid more demand than one full at D-1.
Matched up-gauge pilot
Larger aircraft on selected departures, compared with matched ones.
Decide with a range
The swap decision with a range of revenue outcomes and stated assumptions.

The triage, driver by driver3 minThe three what-ifs through the three questions, and the fitted terms against the truth.

The three what-ifs through the triage Table

What-if	Set by a demand-reacting rule?	Varied in history?	Outcome capped?	Verdict
Fare -5%	Yes	Yes	Partly	Confounded: wrong sign
Capacity 160 -> 180	No	No: always 160	Yes	No information: needs unconstraining and a pilot
Competitor +25%	No	Yes: a 2024-03-01 step on three other routes	Partly	Usable as a forecast under unchanged policy

Fitted terms versus the simulator's truth (per log unit) Chart

Fare: modelpositive: fares followed demand0.5
Fare: truth-1
Competitor capacity: model-0.42
Competitor capacity: truth-0.54

Why the fare term flipsadvanced4 minThe omitted-variable term applied to the simulator's fares, and the full scenario grid.

Where the wrong sign comes from Formula

\hat\beta_{\text{short}} \to \beta + \gamma\,\frac{\operatorname{Cov}(f,\ d)}{\operatorname{Var}(f)}

$\beta$: the true effect of log price on log sales; -1 for fares in the simulator
$f$: log price as set by the pricing rule
$d$: the demand the rule reacted to and the regression left out (known drivers not in the model, the unseen shock)
$\gamma$: the effect of d on sales; the bias is gamma times the slope of d on f

In the simulator log fare = 0.55 x known drivers + 0.5 x an unseen shock; the model controls for some drivers, never the shock. The term is positive whenever prices rise with demand; boosting and partial dependence inherit it, since flexibility creates no variation. It vanishes only if gamma = 0 or price moves independently of demand.

The scenario grid as small multiples Illustration

Change in passengers per flight-day for each fare change, at 160 seats. The model's line rises with the fare (its fare term is +0.50); the simulator's falls (elasticity -1). Every non-zero point of the grid has the wrong sign.

Change in revenue per flight-day. The model's curves for 160 and 180 seats lie on top of each other: it cannot see capacity. The truth separates them, and with 180 seats true revenue keeps rising up to the +10% edge of the grid.

Precomputed by python/precompute/scenario-lab.py on 2026-10-11 (synthetic, seed 42): a pooled log-linear forecasting model fitted on 2022-01-01 to 2024-12-31, against the simulator re-run with only the input changed, IST-TZX, 2025-01-01 to 2025-06-30. Competitor capacity +0%.

The model's answers against the simulator's truth over the whole fare grid, for 160 and 180 seats.

Designs that answer what-ifs3 minForecast versus what-if side by side, and four ways to vary an input honestly.

Forecast versus what-if Comparison

Criterion	Forecast	What-if (causal)
Question	What will happen under current policy?	What would happen if we changed an input?
Data it needs	History of the same process	Variation not chosen by demand or policy
How it is validated	Held-out error on later origins	Experiment, quasi-experiment or a known simulator
Typical failure	A break or regime change	Confounding and censoring read as effects

Four ways to answer a real what-if Comparison

Criterion	Randomized test	Difference-in-differences	Structural model	Labelled scenario
Varies the input by	Design: units (stores, departures, users) assigned at random	A change in some regions or routes, not others	An elasticity taken from a test or study	Nothing: assumed values
Main assumption	No interference between test and control	Parallel trends without the change (Day 29)	The borrowed elasticity transfers	Stated openly, with a wider band
Cost	Revenue at risk during the test	Needs a clean comparison group	Modelling effort; assumptions to defend	Cheap; honest only if labelled

What a wrong what-if costs2 minThe half-year revenue gaps behind each answer.

Half-year revenue gap, truth minus model (EUR thousand, IST-TZX) Chart

Price -5%254 EUR k
Capacity 160 -> 180the model says about zero193 EUR k
Competitor +25%63 EUR k
Capacity 180 and competitor +25%145 EUR k

Today's run over 362 flight-days, synthetic. The gap is what a decision read off the model would misjudge.

Limits of today's numbers1 minFour limits to write next to every number in the defense.

Limits to state next to every number today Overview

One simulator
Known pricing rule, elasticities and cap; a real airline knows none, so each claim needs a test.
Hypothetical costs
Revenue gaps use the simulator's fares, not real accounts.
One model form
A pooled log-linear regression; a richer model changes numbers, not the triage.
One half-year
January-June 2025 on one route; other routes and seasons may differ.

What a staff-level defense addsadvanced2 minFive habits that make a forecasting defense safe to act on.

What a staff-level defense adds Overview

Name the question each number answers
Held-out error certifies the forecast under current policy, nothing more.
Ask how the input was set
Before reading any coefficient, find the rule or person that chose the input's values.
Flag censored rows
Mark capped days in the data, report error on them separately, unconstrain before sizing.
Price the wrong answer
Put the half-year cost of reading the model's what-if next to the cost of a pilot.
Offer the cheapest design
End every 'cannot answer' with the evidence that would answer it and what it costs.

Memory anchor

A forecast is conditional on the policy that made the history: ask how the input was set, whether it varied, and whether the outcome was capped.

Useful for

Preparing the review of any forecasting project: one page of evidence, limits and an explicit 'cannot answer' list.
Responding to scenario requests (price, capacity, competitor, promotion) with a driver-by-driver triage.
Deciding what evidence to collect next when a decision depends on an input that never varied.

Not suitable for

Quantifying causal effects: the table is a teaching comparison in a simulator where the truth is known.
Real routes, stores or portfolios, where the "truth" column does not exist and the triage has to be argued from how the driver was set.

Traps

Accuracy as licence

A credit-risk model's good held-out score is quoted as support for its answer to "what if we raise limits?", and the what-if numbers enter a business case.

How to avoid it

State the question each evaluation answers; a held-out score answers "how well do we predict under current policy".

Reading the cap as demand

A bestseller is out of stock on a third of store-days; the model trained on sales predicts the shelf, and extra stock looks worthless. Sold-out flights do the same to seats.

How to avoid it

Flag capped observations, treat them as lower bounds, and unconstrain before sizing any capacity or price action.

Senior reasoning

A senior defends a forecasting project by listing what it shows and what it cannot, in the same page. They triage each what-if by how the driver was assigned, whether it varied and whether the outcome is censored, and they propose the cheapest design that varies the input. They accept that a defense that volunteers its limits is more persuasive to a technical reviewer and safer for the business.

Interview question

"Your demand model is accurate. The VP wants to use it to choose next quarter's prices. What do you say?" Structure the answer.

Outline your answer first.

Recall cards scheduled after completion

Name the three questions of the what-if triage for a forecasting model's input.
What are the six lines of the forecasting-project defense card?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Causal Inference: What If — Miguel A. Hernán, James M. Robins, 2020core
Why a predictive association does not identify an intervention effect, and what assumptions or designs do. (Part I, ch. 7 Confounding and ch. 8 Selection bias)
Better unconstraining of airline demand data in revenue management systems for improved forecast accuracy and greater revenues — L. R. Weatherford, S. Pölt, 2002core
Methods for unconstraining capacity-limited airline booking data, supporting the point that bookings on closed flights understate demand.
The Theory and Practice of Revenue Management — Kalyan T. Talluri, Garrett J. van Ryzin, 2004optional
Standard revenue-management reference on forecasting, unconstraining and capacity control (chapter mapping not checked here).
Forecasting: Principles and Practice (3rd edition) — Rob J Hyndman, George Athanasopoulos, 2021optional
The forecasting workflow, evaluation on a test set and residual-based intervals that the defense card relies on. (Sections 5.1, 5.8-5.10)
A comparison of forecasting methods for hotel revenue management — Larry R. Weatherford, Sheryl E. Kimes, 2003optional
Pickup and regression methods among the lowest-error hotel arrivals forecasts, a credible revenue-management baseline.

Optional sections

Deep dive · 15 min

What produced the table. python/precompute/scenario-lab.py fits a pooled log-linear regression on all routes for 2022-01-01 to 2024-12-31 (20,824 training rows; in-sample R-squared 0.498 on log passengers): log passengers on route fixed effects, log average fare, weekday, month, holiday, event, trend and log competitor capacity, with forecasts capped at the seat count and a smearing factor of 1.0156 for the log retransformation. It then predicts IST-TZX January-June 2025 under each setting. The truth re-runs the simulator with the same random draws and only the input changed, so the difference is the true effect without simulation noise. The simulator's own facts: a true elasticity of -1.0 on this route, a demand effect of -0.12 in log units for a 25% competitor capacity step, a pricing rule that raises fares with known drivers (weight 0.55) and with an unobserved demand shock (weight 0.5), and a booking cap at 106% of seats.

Why capacity is the hardest. Three things stack: no variation in the input, a capped outcome and a model that predicts the conditional mean. Even a model with the right structure cannot learn the response to 180 seats from data in which sales never exceeded 169. Options are unconstraining historical demand (Weatherford and Polt compare methods; Talluri and van Ryzin cover the revenue-management practice), modelling the cap as censoring in the likelihood, or varying the input by a pilot.

Why price is confounded, in one line. If log fare is $0.55\,s + 0.5\,u$ and $u$ also moves demand, the fare coefficient absorbs $u$ . Day 5's deep dive derives the size of the bias; here the model's +0.50 against -1.00 is that bias at work.

Why competitor capacity looks fine, and where it would not. In the simulator, competitor capacity steps up on three routes on 2024-03-01 and otherwise fluctuates around its base; the airline's pricing rule does not see it as a demand signal. A regression can learn its coefficient from that variation. If competitors responded to our own demand, or if the step coincided with our own fare change, the same regression would be confounded.

Ways to answer real what-ifs. A randomized or matched price test (Day 5); difference-in-differences with a parallel-trends check (Day 5); a structural demand model with the elasticity taken from an experiment and used as an input to the forecasting model; scenario forecasts that are labelled as "if the following assumptions hold" and carry a wider band for the uncertainty in the assumed driver. The last is the honest use of the forecasting model here.

Re-running the grid. cd python && uv run python precompute/scenario-lab.py regenerates the JSON. Quote numbers from the file or from your own run; do not round results into claims about real airlines.

Hands-on lab: Where the baseline is least checkable - slice the held-out error by forecast band · 15 min

Run the query on the sample database. It scores the pickup baseline (D-14 bookings plus the 2023 average pickup for the route and weekday) on all departures from 2024-07-01 to 2024-12-31 and slices the error by how full the baseline expects the flight to be. Read three columns together: the error, the signed bias and the share of departures at the booking cap. Then answer: (1) in which band is the observed error least trustworthy as a measure of the error against demand, and why? (2) the signed bias changes sign across bands; what does that say about a single pickup average per route and weekday, and which slice would you show first in the defense? (3) name one sentence that belongs in the limitations list because of the last column.

WITH dep AS (
  SELECT f.date, f.route, f.weekday, f.bookings_final, b.cumulative_bookings AS bk_d14,
         CAST(f.capacity * 1.06 AS INTEGER) AS cap_bookable
  FROM flight_day AS f
  JOIN booking_curve AS b
    ON b.date = f.date AND b.flight_no = f.flight_no AND b.days_to_departure = 14
), pickup AS (            -- fixed 2023 pickup: every 2023 departure had flown before the 2024 forecasts
  SELECT route, weekday, AVG(bookings_final - bk_d14) AS pu
  FROM dep WHERE date BETWEEN '2023-01-01' AND '2023-12-31' GROUP BY route, weekday
), ev AS (
  SELECT d.*, d.bk_d14 + p.pu AS fc
  FROM dep AS d JOIN pickup AS p ON p.route = d.route AND p.weekday = d.weekday
  WHERE d.date BETWEEN '2024-07-01' AND '2024-12-31'
)
SELECT CASE WHEN fc >= 0.95 * cap_bookable THEN '3 forecast >= 95% of cap'
            WHEN fc >= 0.80 * cap_bookable THEN '2 forecast 80-95% of cap'
            ELSE '1 forecast < 80% of cap' END AS forecast_band,
       COUNT(*) AS departures,
       ROUND(AVG(ABS(bookings_final - fc)), 2) AS mae_seats,
       ROUND(AVG(bookings_final - fc), 2) AS mean_actual_minus_fc,
       ROUND(AVG(bookings_final >= cap_bookable), 3) AS share_at_cap
FROM ev GROUP BY 1 ORDER BY 1;

Interview practice · 10 min

A product team shows a model that predicts weekly subscriptions well and asks you to use it to decide how much to raise the monthly price. Walk through your answer.

Answer as you would in the interview.

Content version 6 · last verified 2026-10-18 · depends on numpy, sqlite

1 Recall · 5 min

2 Concept · 10 min