Core day 5 of 60 · Week 1 · Part I · application depth

Causal Thinking and Experiments

The commercial director asks: "If we cut IST-AYT fares 5% next quarter, how many more seats will we sell?" In your history, high-fare departures carry more bookings, and your demand forecast is accurate. How can both be true while the forecast gives the wrong answer - and what evidence would answer the question?

By the end: You draw the causal diagram behind an observed association, name the confounder, show why adjusting for recorded variables may not remove it, and propose a defensible experiment or a qualified observational design.

At a glance

When the treatment follows demand, history shows the pricing rule, not the price effect; draw the diagram, then randomize or state the assumptions of a quasi-experiment.

Observe vs intervene
Coupon holders order more; that is not what a coupon does. Same for fares: seeing is not setting.
Confounder
Demand drives both fare and bookings, so the historical slope can flip sign.
Adjust with care
Compare within strata of every confounder; unmeasured ones keep the bias, colliders add new bias.
Randomize
A random +/-5% fare recovers -1.37 against a true -1.35.
Fit is not truth
The confounded model fits history better than the true one.

Recall · 5 min

From Day 4: why can a valuable effect be non-significant, and what does peeking 20 times do to the false-positive rate of an A/A test?

Open due reviews

Concept · 10 min

Revenue management raises fares when it expects or sees strong demand, and strong demand also brings bookings. So in history high fares and high bookings travel together, although a higher fare sells fewer seats. Demand is a Confounder. A forecast answers "what do we see when fares are high"; a pricing decision needs "what happens if we set them", the causal effect $P(Y \mid do(X))$ .

Fares and bookings rise together in history because demand drives both - intervening on fare reveals the negative effect

Predict first

RM raises fares on high-demand days. The true effect of fare on bookings is negative. What will a regression of bookings on fare in history most likely show?

Text description of the figure

A simulation with a confounder (demand) that affects both treatment (fare) and outcome (bookings). Panels show the pooled scatter with its fitted slope, the same data stratified by the confounder, and the result of setting the fare by intervention. A selection filter keeps only some units to show selection distortion.

Two questions that sound alike Comparison

Criterion	Observing	Intervening
Question	On departures with high fares, how many bookings did we see?	If we set the fare 5% lower, how many bookings would we get?
Answered by	A forecast trained on history, and it answers well	An experiment, or adjustment for every confounder
In today's simulation	Fare coefficient +0.46: high fares, many bookings	Elasticity -1.35: a cut sells about 7% more seats

The causal diagram of airline pricing Diagram

Season and holidaysrecorded
- back door: to Bookings
RM prices
Fareset by revenue management
- effect < 0: to Bookings
Booking paceNOT recorded
- RM reacts: back to Fare
back door
Bookingsthe outcome

A back door is any path that starts with an arrow INTO fare. Season, holidays and pace raise both fare and bookings; adjusting for season closes one back door, the unrecorded pace keeps the other open. Read the adjustment set off the diagram, not off the column list.

One history, four views: pooled, by season, randomized, filtered Illustration

History, pooled: slope +0.35 bookings per EUR. High fares sit on high-demand days.

Same points split by season: off-peak -0.54, shoulder -0.61, peak -0.60. Within each season a higher fare sells fewer seats.

Fares set at random (do(fare)): slope -0.57, close to the simulator's true -0.60. Randomization cut the arrow from demand into fare.

Where it breaks: keep only flights at least 80% full and the same experiment shows -0.23. Filtering on the outcome flattens the effect.

off-peak
shoulder
peak
capacity 180 (dashed)

Synthetic: 600 departure days per panel from the seeded confounding-lab generator (true effect -0.60 bookings per EUR, confounder strength 1). Slopes are least-squares fits on the points shown.

The pooled slope reflects the pricing rule; within seasons and under randomization the price effect is negative. Filtering on the outcome distorts even an experiment.

Today's run: four estimates of one elasticity Chart

History, fare onlywrong sign0.46
History, adjustedrecorded drivers only; still wrong sign0.25
Randomized fare test-1.37
Truth (simulator)-1.35

Bar length shows size; the value shows the sign. Log-log coefficients from 2,000 simulated departure days on one route.

The director's question: bookings after a 5% fare cut Chart

Confounded forecast modelpredicts a loss-1.2 %
Randomized experiment7.3 %
Truth (simulator)7.2 %

Same data world, opposite decisions. Fare is an excellent proxy for demand, so the better a forecast fits history the more its fare coefficient reflects the pricing rule. Answer what-ifs with an experiment or a stated causal design (Day 28).

Which design answers a what-if? Diagram

Can you randomize?
- yes: to Randomized test
no
Comparable controls?
- yes: to Difference in differences
no
Interrupted time seriesa break at a known date
- then, for both: to State the assumptions
Randomized testpower, guardrails, no peeking
Difference in differencesassumes parallel trends
State the assumptionsplus a sensitivity check

Every observational design trades the randomization guarantee for an assumption you must state and defend.

Draw the causal diagram first; every path into fare is a back door that mixes demand into the price coefficient.
Adjust only for measured confounders; never for a Collider or an outcome. Watch for selection bias and Simpson's paradox.
Randomize when you can; otherwise use difference-in-differences and its relatives, and state their assumptions.

Assumptions

No interference between units - a cheaper fare on one flight can pull passengers from the same day's other flights, so randomize at a level that contains the spillover.
For observational adjustment, every confounder on a back-door path is measured and adjusted for; this is untestable from data alone.
For difference-in-differences, treated and control units would have moved in parallel without the intervention, and nothing else changed for the treated units at the same time.

Application and workplace decision · 10 min

Run the simulation of one leisure route (synthetic, built like the capstone simulator: true elasticity -1.35; RM raises fares with recorded demand drivers and with an unrecorded booking pace). It estimates the fare effect:

from history;
from history adjusted for recorded drivers;
from a randomized +/-5% fare test;

then answers the director's what-if with each. Predict the sign of the historical estimate first.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np

rng = np.random.default_rng(5)
n, elasticity = 2_000, -1.35  # departure days on one leisure route; TRUE price elasticity (synthetic)
signal = rng.normal(0, 0.30, n)  # known demand drivers: season, weekday, holidays (recorded)
u = rng.normal(0, 0.08, n)       # early-booking pace the RM analyst reacts to (NOT recorded)

def bookings(log_p):
    return rng.poisson(np.exp(np.log(130) + signal + elasticity * log_p + u))

def ols(y, *xs):  # returns coefficients and residual variance
    X = np.column_stack([np.ones(len(y)), *xs])
    beta = np.linalg.lstsq(X, y, rcond=None)[0]
    return beta, np.var(y - X @ beta)

# History: RM raises fares when demand is expected (signal) or seen (u) to be high, as in airline_sim.py
log_p = 0.55 * signal + 0.50 * u + rng.normal(0, 0.02, n)
y = np.log(bookings(log_p))
print(f"corr(log fare, log bookings) = {np.corrcoef(log_p, y)[0, 1]:+.2f}")
print(f"fare coefficient, fare only: {ols(y, log_p)[0][1]:+.2f}")
b_obs, v_obs = ols(y, log_p, signal)
_, v_true = ols(y - elasticity * log_p, signal)  # the same model with the TRUE elasticity imposed
print(f"fare coefficient, adjusted for recorded drivers: {b_obs[1]:+.2f} (true {elasticity:+.2f})")
print(f"R2 on history: confounded model {1 - v_obs / np.var(y):.2f} | causal model {1 - v_true / np.var(y):.2f}")

# Experiment: on each day a coin flip moves the fare 5% up or down around the RM price
coin = rng.choice([-1.0, 1.0], n) * np.log(1.05)
b_exp, _ = ols(np.log(bookings(log_p + coin)), coin, signal, log_p)
print(f"randomized price test estimate: {b_exp[1]:+.2f}")

# What-if: cut every fare by 5%
for name, e in [("confounded model", b_obs[1]), ("experiment", b_exp[1]), ("truth", elasticity)]:
    print(f"{name:16s} predicts {np.expm1(e * np.log(0.95)) * 100:+.1f}% bookings")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 41 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

Potential outcomes: what a causal effect is6 minPotential outcomes, the ATE and ATT defined as math, an eight-customer coupon table worked by hand, and the naive comparison split into effect plus selection.

Potential outcomes: the object a causal question asks about Formula

\begin{aligned} Y_i &= T_i\,Y_i(1) + (1 - T_i)\,Y_i(0) \\ \tau_i &= Y_i(1) - Y_i(0) \\ \text{ATE} &= \mathbb{E}[\tau_i], \quad \text{ATT} = \mathbb{E}[\tau_i \mid T_i = 1] \end{aligned}

$Y_i(1),\ Y_i(0)$: unit i's outcome with and without the treatment; both are defined for everyone, only one is ever observed
$T_i$: treatment actually received: 1 = got the coupon, the call, the price change
$Y_i = T_i Y_i(1) + \dots$: consistency: what we record is the potential outcome of the treatment received (needs one version of treatment and no interference)
$\tau_i$: the individual effect; never observed for anyone, which is the fundamental problem of causal inference
$\text{ATE}$: average effect over the whole population: the estimand for 'give it to everyone'
$\text{ATT}$: average effect over those who were treated: the estimand for 'was it worth it for the ones who got it'

The do-notation of the concept section names the same object: E[Y | do(T = t)] = E[Y(t)]. Only one potential outcome is ever recorded per unit; every causal method fills in the missing column with an assumption, and that assumption is what you must defend.

Worked by hand: a coupon sent to loyal customers (hypothetical retailer) Table

Customer	Coupon?	Y(0): orders without	Y(1): orders with	Effect	What the data show
A	yes	4	5	1	5 (Y(0) unseen)
B	yes	3	4	1	4 (Y(0) unseen)
C	yes	5	5	0	5 (Y(0) unseen)
D	yes	2	3	1	3 (Y(0) unseen)
E	no	1	2	1	1 (Y(1) unseen)
F	no	0	1	1	0 (Y(1) unseen)
G	no	2	2	0	2 (Y(1) unseen)
H	no	1	3	2	1 (Y(1) unseen)
All 8	mixed	2.25	3.125	ATE 0.875	never computable from data
Coupon group A-D	yes	3.5	4.25	ATT 0.75	observed mean 4.25
No coupon E-H	no	1.0	2.0	ATC 1.0	observed mean 1.0

Orders next month. The CRM team chose its four most loyal customers. Naive comparison 4.25 - 1.0 = 3.25 orders, almost four times the true average effect. Verified in Python; only a simulation ever sees both columns.

The naive comparison, decomposed: effect plus selection Formula

\def\E{\mathbb{E}} \begin{aligned} &\underbrace{\E[Y \mid T=1] - \E[Y \mid T=0]}_{\text{naive}} \\ = &\underbrace{\E[\tau \mid T=1]}_{\text{ATT}} \\ + &\underbrace{\E[Y(0) \mid T=1] - \E[Y(0) \mid T=0]}_{\text{selection}} \end{aligned}

$\text{naive}$: the only contrast the data give directly: 4.25 - 1.0 = 3.25 orders in the table
$\text{ATT}$: what the coupon did for those who got it: 0.75
$\text{selection bias}$: how differently the two groups would have behaved WITHOUT treatment: 3.5 - 1.0 = 2.5 orders
$T \perp (Y(0), Y(1))$: randomization makes the groups exchangeable: the bias term is zero and the ATT equals the ATE

Derivation: add and subtract E[Y(0) | T = 1], then use consistency. Every observational design is an argument for why the selection term is zero, or how it was removed.

The coupon table's naive gap is mostly selection Bridge

Naive difference (4.25 - 1.0)3.25 orders
Selection bias (3.5 - 1.0)loyal customers order more anyway−2.5 orders
Effect on the treated (ATT)0.75 orders

Naive = ATT + selection, exactly: 3.25 - 2.5 = 0.75 orders. The CRM team picked customers who already order more, so three quarters of the gap is who got the coupon, not what it did.

The decision picks the estimand (coupon table) Hierarchy

Which decision?the same eight customers
- coupon for everyoneATE = 0.875average over all 8
- keep the targetingATT = 0.75average over the 4 who got it
- extend to the restATC = 1.0average over the 4 who did not

The three differ whenever effects differ across units. Name the estimand before choosing a method; a design that identifies one need not identify the others.

Randomization: why a coin flip removes the selection term4 minThe two-line proof that randomization makes the naive difference unbiased, all 70 assignments of the coupon table enumerated, and why precision still needs n.

Why a coin flip removes the selection term, in two lines Formula

T \perp (Y(0), Y(1)) \;\Rightarrow\; \mathbb{E}[Y \mid T{=}t] = \mathbb{E}[Y(t)]

$\mathbb{E}[Y \mid T{=}1] = \mathbb{E}[Y(1) \mid T{=}1]$: consistency: the treated show their treated outcome
$= \mathbb{E}[Y(1)]$: independence: the coin knows nothing about Y(1), so the treated are a random half; the same holds for T = 0
$\text{naive} = \text{ATE}$: subtract the two lines: the selection term of the decomposition above is zero

Randomization also balances confounders nobody measured or thought of. No adjustment method has that property, which is why experiments are the reference design.

Randomization in potential outcomes: unbiased on average, precise only with n Illustration

All 70 assignments of the coupon to 4 of 8 customers. Their average difference is 0.88, exactly the ATE of 0.88. The CRM choice (the 4 most loyal) gives 3.25; 1 of 70 assignments reach that far.

1,000 random half-splits of 400 customers: mean 0.87 against a true ATE of 0.86, spread (sd) 0.19. Coupons to the most loyal half: 2.86, as far off as with 8 customers.

Left: the lesson's hypothetical eight-customer table, enumerated exactly. Right: 400 synthetic customers, orders without the coupon Poisson with a rate rising in loyalty, individual effects 0, 1 or 2 orders, seeded. Only a simulation shows both potential outcomes.

Every dot is one possible coin flip. Their average is the true effect, whatever the sample size; a hand-picked assignment sits at the edge, and more data does not move it back. With the 8 coupon customers, single random assignments range from -1.5 to +3.25 orders: unbiased is not precise, and precision comes from n (Day 4).

The back-door criterion and the adjustment formula, worked6 minWhich variables to condition on, read from the diagram; the adjustment formula with its two assumptions; and a hospital programme where pooled data reverse the sign.

The causal diagram of airline pricing Diagram

Season and holidaysrecorded
- back door: to Bookings
RM prices
Fareset by revenue management
- effect < 0: to Bookings
Booking paceNOT recorded
- RM reacts: back to Fare
back door
Bookingsthe outcome

The back-door adjustment formula Formula

\begin{aligned} &Y(t) \perp T \mid S \text{ and } 0 < P(T{=}1 \mid S) < 1 \\ &\Rightarrow\; \mathbb{E}[Y(t)] = \textstyle\sum_{s} \mathbb{E}[Y \mid T{=}t, S{=}s]\, P(S{=}s) \end{aligned}

$S$: measured variables that block every back-door path from T to Y and contain no descendant of T (the back-door criterion)
$Y(t) \perp T \mid S$: conditional exchangeability: within a stratum of S, treatment is as good as random
$0 < P(T{=}1 \mid S) < 1$: positivity (overlap): every stratum has treated and untreated units, or its contrast does not exist in the data
$\mathbb{E}[Y \mid T{=}t, S{=}s]$: an observed stratum mean: estimable without any causal assumption
$P(S{=}s)$: population weights give the ATE; the treated units' weights give the ATT

Compare within strata, then average. Regression, matching and inverse-probability weighting compute the same sum; none can repair an S that misses a confounder. Tabulate treatment by stratum first: where a stratum has no untreated units, any model's contrast there is extrapolation, not estimation.

Worked by hand: follow-up calls after discharge (hypothetical hospital) Table

Stratum (patients)	Called: readmitted	Not called: readmitted	Difference
Low risk (500 of 900)	5 of 100 = 5%	40 of 400 = 10%	-5 pp
High risk (400 of 900)	60 of 300 = 20%	30 of 100 = 30%	-10 pp
Pooled, naive	65 of 400 = 16.25%	70 of 500 = 14%	+2.25 pp: calls look harmful
Adjusted, population weights	5/9 x 5% + 4/9 x 20% = 11.7%	5/9 x 10% + 4/9 x 30% = 18.9%	ATE -7.2 pp
Adjusted, called patients' weights	1/4 x 5% + 3/4 x 20% = 16.25%	1/4 x 10% + 3/4 x 30% = 25%	ATT -8.75 pp

Nurses called mostly high-risk patients, so the pooled rates mix risk into the programme (Simpson's paradox). Within each stratum calls help. Verified in Python; valid only if risk is the only confounder.

Why adjustment failed in the codeadvanced4 minThe omitted-variable formula for the simulator, simulated bias against the formula, and why a disciplined pricing team makes it worse.

Back doors in numbers, and why disciplined pricing makes them worse Illustration

Two back doors run from fare to bookings through demand. Adjusting for the recorded drivers closes one; the unrecorded pace keeps the other open.

Same simulated history, four designs. Closing every back door (possible only because the simulator records u) or randomizing lands near the true -1.35; the fully adjusted estimate is noisier, because little fare variation is left once s and u are fixed.

Where it breaks: the bias left after adjusting for s grows as RM prices more consistently. A disciplined pricing rule leaves almost no independent fare variation to learn from.

Synthetic, mirroring the Python block: 2,000 departure days, Poisson bookings (normal approximation above a rate of 50), seeded. The right panel re-simulates the history at six noise levels; the curve is the omitted-variable formula.

The simulator of the Python block: the diagram, the fare coefficient under four designs, and the bias left after adjusting for recorded drivers as the pricing noise shrinks, simulated against the omitted-variable formula.

The bias left after adjusting for recorded drivers Formula

\hat\beta_{\text{adj}} - \beta = \frac{\operatorname{Cov}(u, \log p \mid s)}{\operatorname{Var}(\log p \mid s)} = \frac{0.5\,\sigma_u^{2}}{0.25\,\sigma_u^{2} + \sigma_\varepsilon^{2}}

$u$: the unrecorded booking pace: raises bookings directly and moves the fare by 0.5 u
$\sigma_\varepsilon$: independent pricing noise: the only fare variation not driven by demand
$\sigma_u = 0.08,\ \sigma_\varepsilon = 0.02$: the Python block: bias about +1.6, turning -1.35 into about +0.25

The omitted-variable formula for this simulator. As the pricing noise shrinks, the denominator is dominated by u and the bias approaches +2. More recorded controls cannot help: the bias lives in the fare variation that only u explains.

A randomized fare change is an instrument: it reaches bookings only through fare Diagram

Coin flip: +/-5%set by design
moves
FareRM price plus the flip
- effect -1.35: to Bookings
Booking paceNOT recorded
- RM reacts: back to Fare
back door
Bookingsthe outcome

An instrument changes the treatment and affects the outcome only through it. The flip-induced fare variation is independent of pace, so it identifies the effect despite the open back door; exclusion (no coin-to-bookings arrow) holds by design, which is why the randomized test returned -1.37.

What to adjust for, and what never to3 minConfounders, unmeasured confounders, mediators, colliders and pure predictors of the outcome, with the four ways a comparison goes wrong.

What to adjust for, and what never to Table

Variable	Example	Adjust?	Why
Confounder	season (pricing), patient risk score (healthcare)	yes, if measured	closes a back door
Unmeasured confounder	booking pace, a clinician's judgement	cannot	the path stays open: randomize or find a design
Mediator	coupon redemption page visits (retail)	no, for the total effect	removes part of the effect you want
Collider or outcome	load factor; being admitted to hospital; loan still open	no	opens a new non-causal path or filters on the outcome
Pre-treatment predictor of the outcome only	store size, last year's spend	optional	no bias either way; can reduce variance

Draw the diagram first, then read the adjustment set off it (Hernan and Robins ch. 7-8; Cunningham ch. 3).

Four ways a comparison goes wrong Overview

Confounding
A common cause of treatment and outcome: demand behind fare and bookings.
Collider
Studying only admitted patients, or only funded loans, conditions on a common effect.
Selection
Only customers who stayed, or machines still running, distort the comparison.
Simpson's paradox
A pooled trend reverses within groups; the diagram says which is causal.

Difference-in-differences: the 2x2 worked, then as a regression6 minParallel trends written in potential outcomes, a home-energy programme worked by hand, the same estimate as an interaction coefficient, and what breaks it.

Difference-in-differences Formula

\hat\tau = (\bar Y_{T,\text{after}} - \bar Y_{T,\text{before}}) - (\bar Y_{C,\text{after}} - \bar Y_{C,\text{before}})

$T$: routes that received the fare cut
$C$: comparable routes without it
$\hat\tau$: the effect, if T would have moved like C without the cut (parallel trends)

The second difference removes shared shocks such as season. It cannot remove something that hit only the treated routes at the same time.

Parallel trends in potential outcomes, and DiD as a regression Formula

\def\E{\mathbb{E}} \begin{aligned} &\E[Y_1(0)-Y_0(0) \mid G=T] \\ = &\E[Y_1(0)-Y_0(0) \mid G=C] \Rightarrow \hat\tau \to \text{ATT} \\ &Y_{gt} = \alpha + \beta\text{Tr}_g + \gamma\text{Post}_t + \tau\text{Tr}_g\text{Post}_t \end{aligned}

$Y_t(0)$: untreated potential outcome in period t (0 = before, 1 = after); for the treated group after the change it is never seen
$=$: parallel trends: without treatment, the treated group would have changed as much as the control group did
$\to \text{ATT}$: DiD estimates the effect on the treated units, not on everyone
$\alpha,\ \beta,\ \gamma$: control level before, fixed gap between the groups, change common to both
$\tau$: coefficient of the interaction: exactly the 2x2 difference-in-differences

The assumption is about a counterfactual change, so data can only probe it indirectly: pre-period trends and placebo dates. Repeated observations of the same units are correlated, so cluster standard errors by unit; with few treated clusters inference is fragile (Day 29).

Worked by hand: home-energy reports in one city (hypothetical utility) Table

Group	Before (Jan-Mar)	After (Apr-Jun)	Change
Households sent reports	30.0	27.2	-2.8: report effect plus spring
Comparable city, no reports	28.0	26.4	-1.6: spring alone (the counterfactual change)
Treated minus control	2.0: fixed gap	0.8	DiD = -2.8 - (-1.6) = -1.2
Regression coefficients	alpha = 28.0, beta = 2.0	gamma = -1.6	tau = -1.2

Average daily kWh per household. Before-after alone (-2.8) credits the reports with the weather; the after-only gap (0.8) carries the fixed difference between cities. The -1.2 is the ATT for these households; it says nothing about reports in another city. Verified in Python.

Difference-in-differences: right when trends are parallel, wrong when not Illustration

DiD estimate -4.3 pts against a true -5.0 pts. Placebo break at week 15 (pre-period only): 0.6 pts, consistent with parallel trends.

DiD estimate -7.4 pts against a true -5.0 pts. Placebo break at week 15 (pre-period only): -1.2 pts, a warning that the groups were not moving together.

Synthetic weekly rates (any share metric: retention, conversion, load factor): a shared seasonal curve, treated group 3 points higher, noise 1.2 points, true effect -5 points from week 31; the right panel adds a drift of -0.12 points per week to the treated group only (seeded).

Dashed: the treated group's assumed untreated path, the control series shifted by the pre-period gap. When the treated group was already drifting, the same arithmetic credits the drift to the intervention, and a placebo break inside the pre-period exposes it.

Is this difference-in-differences credible? Diagram

Pre-trends parallel?
- no: to Rethink the control group
yes
Placebo dates near 0?
- yes: to Anything else changed?
no
Rethink the control groupor model the trend
Anything else changed?
- no: to Report the estimate
yes
Name the rival eventa second change on the same date
Report the estimatewith its assumption

Pre-period checks test the parallel-trends assumption; nothing in the data rules out a different change on the same date.

Every assumption as math, each with its diagnostic4 minNo interference, consistency, exchangeability, positivity and parallel trends, how each fails in practice and the check that probes it.

Each causal assumption, as math, with the check that probes it Table

Assumption	As math	Fails when	Diagnostic or remedy
No interference	Y_i depends only on T_i	a discount on one product steals sales from another; a promoted store pulls from its neighbour	randomize clusters (stores, regions, days) that contain the spillover
Consistency	Y_i = Y_i(T_i)	the treatment has versions: a follow-up call by a nurse or by a bot	define the treatment precisely, one version per arm
Exchangeability	Y(t) independent of T	assignment followed the outcome's drivers: loyalty, risk, demand	randomize; otherwise rely on the conditional version
Conditional exchangeability	Y(t) independent of T given S	an unmeasured confounder: a clinician's judgement, a booking pace	untestable: sensitivity analysis, negative-control outcomes
Positivity (overlap)	0 < P(T = 1 given S) < 1	some strata are always treated, such as a credit-limit offer only for top-tier accounts	plot the propensity overlap; trim or redefine the population
Parallel trends	equal untreated changes in both groups	the treated group was already drifting, or another change hit it on the same date	pre-trend plot, placebo dates, a second control group

Randomization buys the first three rows by design; every observational method replaces them with an assumption you must state and probe.

Which assumptions data can check, and which they cannot Scale

Positivitytabulate treatment by stratum; plot propensity overlap
Parallel trendsprobed only: pre-trend plot, placebo dates
No interferencea design question: randomize clusters (cities, weeks)
Exchangeability given Ssensitivity analysis: how strong must a hidden confounder be?

Only positivity and pre-trends show up in data. Interference is settled by design: a telecom retention offer that spreads by word of mouth, or a ride-hailing price shown to drivers and riders at once, needs clusters as the unit. For exchangeability, report how strong an unmeasured confounder would have to be to erase the effect.

Why better fit can mean a worse answer2 minThe confounded model wins on R2 and loses on the decision.

Fit on history picks the wrong model Chart

Confounded modelfare as a proxy for demand0.41 R2
Model with the true elasticity0.06 R2

Share of variance in log bookings explained on the simulated history. Better fit, wrong causal answer.

Designing the fare experiment3 minUnit, treatment, size, guardrails and limits.

A randomized fare test that the business can accept Sequence

Unit
Departure day on one route; check spillover to same-day flights.
Treatment
A coin flip moves the RM fare 5% up or down.
Size and length
Sized for the smallest elasticity that changes the decision (Day 4).
Guardrails
Revenue per departure and load factor, with stop rules agreed in advance.
Limits
Check fairness and legal limits on price variation first.

A short test measures the short-run elasticity; competitor reactions and repeat-customer learning are outside it.

Other designs when you cannot randomizeadvanced3 minInterrupted time series, synthetic control, adjustment and Granger tests, each with the assumption it trades for randomization.

Designs when you cannot randomize Table

Design	Compares	Key assumption
Difference-in-differences	change in treated vs change in comparable untreated units	parallel trends without the intervention
Interrupted time series	the series before and after a known date	nothing else broke at that date
Synthetic control	a treated unit vs a weighted mix of untreated ones	the mix tracks the unit before treatment
Regression adjustment or matching	treated vs untreated units with the same measured confounders	every confounder is measured (untestable)
Granger causality	whether one series' past helps predict another	predictive precedence only, not an effect

Synthetic control suits one treated unit with many untreated donors: a region that changed its tariff (energy), a single store format (retail). Its credibility is the pre-period fit plus placebo runs on the donors.

Memory anchor

Ask how the treatment was assigned: when price follows demand, history answers 'what we saw', not 'what if'.

Useful for

Pricing (retail, travel), promotions and retention offers (marketing, telecom), care programmes (healthcare) and policy changes (energy tariffs, credit limits) phrased as "what if we change X".
Reviewing claims that a feature, coefficient or dashboard correlation "drives" an outcome.
Choosing between an experiment and a quasi-experimental design, and writing down its assumptions.

Not suitable for

Pure forecasting under an unchanged policy - there, a confounded but stable association is legitimately useful.
Questions where no variation in the treatment exists and none can be created; no method recovers an effect from data that never varied.

Traps

Using the forecasting model for a what-if

A demand model fits history well, so its price coefficient or partial dependence is used to price: an online shop's markdowns, a bank's deposit rates, today's fares. The sign can be wrong because prices were set in response to demand.

How to avoid it

Ask how the treatment was assigned; for what-if questions use an experiment or an explicit causal design.

Controlling for everything available

Adding post-treatment variables (load factor after a fare change; card utilisation after a credit-limit increase) or filtering on outcomes (only accounts still open) creates new bias instead of removing it.

How to avoid it

Draw the diagram first; adjust for confounders only, never for mediators, colliders or post-treatment variables.

Senior reasoning

A senior first asks how the treatment was assigned. If a person or an algorithm chose it with knowledge of the outcome's drivers, they assume confounding until a design rules it out. They draw the diagram with the business, prefer randomization when it is ethical and affordable, state the assumptions of any observational fallback in plain words, and keep the forecasting model for forecasting. They also tell the director what the experiment cannot see: competitor reactions and long-run effects. The same discipline covers a coupon aimed at loyal shoppers, a nurse call aimed at high-risk patients and a limit increase aimed at good payers: whoever chose the treatment knew the outcome.

Interview question

"Our data show that customers who received discounts spent more. Should we give more discounts?" How do you answer?

Outline your answer first.

Recall cards scheduled after completion

Why do airline fares and bookings correlate positively in history although the true price elasticity is negative?
Name the designs for a causal question and the key assumption of each.

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Causal Inference: What If — Miguel A. Hernán, James M. Robins, 2020core
Randomized experiments as the reference design, the structure of confounding (back-door paths) and of selection bias (conditioning on common effects). (Part I, ch. 2 Randomized experiments, ch. 7 Confounding, ch. 8 Selection bias)
Causal Inference: The Mixtape — Scott Cunningham, 2021optional
Accessible treatment of DAGs, back-door paths and colliders, and of difference-in-differences with its parallel-trends assumption. (Ch. 3 Directed Acyclic Graphs; ch. 9 Difference-in-Differences)
Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing — Ron Kohavi, Diane Tang, Ya Xu, 2020optional
Practical design of online controlled experiments and the observational alternatives when randomization is not possible. (Ch. 11 Observational Causal Studies; ch. 14 Choosing a Randomization Unit)
Investigating Causal Relations by Econometric Models and Cross-spectral Methods — C. W. J. Granger, 1969optional
Defines Granger causality as improvement in prediction from another series' past - predictive precedence, not an intervention effect.
Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California's Tobacco Control Program — Alberto Abadie, Alexis Diamond, Jens Hainmueller, 2010optional
Introduces the synthetic control method for a single treated unit, named here as an option when no randomized or parallel control group exists.

Optional sections

Deep dive · 10 min

Why adjustment failed in the code. Write log fare as $0.55\,s + 0.5\,u + \varepsilon$ and log demand as $s - 1.35 \log p + u$ . Holding the recorded drivers $s$ fixed, fare still moves with $u$ , and $u$ moves bookings directly. The bias of the adjusted coefficient is $\text{Cov}(u, \log p \mid s)/\text{Var}(\log p \mid s) = 0.5\,\sigma_u^2 / (0.25\,\sigma_u^2 + \sigma_\varepsilon^2) \approx +1.6$ , which turns -1.35 into about +0.25. The smaller the independent pricing noise, the worse the bias - exactly the situation of a disciplined RM team.

Instruments. A variable that moves price but affects bookings only through price (a randomized fare perturbation is the cleanest case; a fuel-surcharge change might be another, if it does not signal anything about demand) identifies the effect by instrumental variables. The exclusion assumption is the weak point and cannot be tested from data.

Censoring adds a second distortion. Bookings stop at capacity, so on the highest-demand days the outcome is capped (Day 28). Even a randomized test measures the effect on bookings, which understates the effect on demand for flights that sell out in both arms.

Selection and survivorship. Studying only routes that are still flown, or only customers who stayed, filters on an outcome. Associations in the survivors need not hold in the population that entered.

Granger, ITS and synthetic control. Granger tests ask whether past X improves prediction of Y - useful for forecasting, silent on intervention. Interrupted time series needs a sharp intervention date and no simultaneous change. Synthetic control weights untreated units (for example other routes) to reproduce the treated unit's pre-period path; its credibility rests on that pre-period fit and on placebo tests.

Hands-on lab: A natural experiment in the sample database · 15 min

Competitor capacity on IST-ADB, IST-AYT and IST-FRA steps up 25% on 2024-03-01 (a documented structural break in the synthetic data). Treat those routes as "treated" and the other domestic and short-haul routes as controls. Compare bookings per seat in March-December 2023 with March-December 2024 (same months, so seasonality cancels). Run the query, then compute (1) the naive before-after change for treated routes and (2) the difference-in-differences estimate. The control routes grew by about 3.2 points of capacity, so the before-after drop of about 4.8 points understates the effect; DiD gives about -8.1 points. Then answer: what would violate parallel trends here (for example, a fare policy change on the treated routes at the same time)? The simulator reduces log demand by 0.12 for a 25% competitor step before capacity caps - is your estimate in a plausible range, and why might it be smaller?

WITH periods AS (
  SELECT CASE WHEN route IN ('IST-ADB', 'IST-AYT', 'IST-FRA') THEN 'treated' ELSE 'control' END AS grp,
         CASE WHEN date BETWEEN '2023-03-01' AND '2023-12-31' THEN 'before' ELSE 'after' END AS period,
         bookings_final, capacity
  FROM flight_day
  WHERE market <> 'intl-long'
    AND (date BETWEEN '2023-03-01' AND '2023-12-31' OR date BETWEEN '2024-03-01' AND '2024-12-31')
)
SELECT grp,
       ROUND(AVG(CASE WHEN period = 'before' THEN 1.0 * bookings_final / capacity END), 4) AS before_bps,
       ROUND(AVG(CASE WHEN period = 'after'  THEN 1.0 * bookings_final / capacity END), 4) AS after_bps,
       ROUND(AVG(CASE WHEN period = 'after'  THEN 1.0 * bookings_final / capacity END)
           - AVG(CASE WHEN period = 'before' THEN 1.0 * bookings_final / capacity END), 4) AS change
FROM periods GROUP BY grp ORDER BY grp DESC;

Interview practice · 10 min

A SaaS product team shows that accounts that turned on the new in-app onboarding assistant had 6% higher expansion revenue than accounts that did not (illustrative). They want to make it the default for every new account. Respond.

Answer as you would in the interview.

Content version 6 · last verified 2026-10-11 · depends on numpy, sqlite

1 Recall · 5 min

2 Concept · 10 min