Core day 10 of 60 · Week 2 · Part I · evaluation depth

Analytics Checkpoint

A one-page memo says flights that received a marketing email had a load factor 8.1 points higher (p < 0.001) and recommends emailing every flight. Before it goes to the commercial director, which parts of the evidence survive a senior review - grain, comparison, uncertainty, chart - and what would you recommend instead?

By the end: You audit a short analysis end to end: find its grain error, separate the confounded comparison from a defensible one, read the interval correctly, critique its chart, and propose the next piece of evidence.

At a glance

Review any analysis in the order errors compound (table, grain, comparison, uncertainty, chart, boundary); today an 8-point email effect shrinks to under one.

Order matters
A later check is meaningless on a table that fails an earlier one - in a claims memo or a churn deck alike.
Grain
Compare one row per flight, not per email: the gap moves from 8.14 to 6.75.
Fair comparison
Emails follow season; within season the gap is +0.30 +/- 0.51.
Read the interval
Rules out effects above about 0.8 points; does not prove zero.
Honest chart
A bar axis at 0.75 draws an 11% difference as a 7x one.

Recall · 5 min

Checkpoint warm-up, no notes, two minutes each: (1) what must happen before any preprocessing step is fit? (2) A join multiplies rows by 13 uniformly - which metrics survive? (3) Name the one EDA check that reversed the IST-TZX March trend. Then clear your due review cards.

Open due reviews

Concept · 10 min

Week 2 gave you the checks a senior runs on any analysis, in the order errors compound: table, grain, comparison, uncertainty, chart, boundary. Each check can change the answer on its own, and a later check is meaningless on a table that fails an earlier one: a perfect interval on a fan-out join is a precise estimate of the wrong number.

When treatment follows the confounder, the raw gap measures the targeting rule - shown here with fares and bookings, the same structure as the email case

Predict first

If marketing had emailed a random 40% of flights, what would the raw emailed-vs-not gap estimate?

Text description of the figure

The confounding lab uses fares and bookings with season as the confounder, not email; the structure is the same as the memo: the "treatment" is set more often where the outcome is already high. The pooled slope can turn positive although the true effect is negative, stratifying by season brings each within-season slope back near the truth, and a do() switch sets the treatment at random. A selection filter shows how keeping only some units distorts it.

Six checks, in the order errors compound Sequence

Is the table right?
Coverage, keys, integrity, missingness (Day 6).
Is the grain right?
One row per unit of analysis before comparing or summing (Days 2, 7).
Is the comparison fair?
Who received the treatment, and why: confounding and selection (Day 5).
How sure are we?
An effect size with its interval, not a p-value alone (Day 4).
Does the chart say only that?
Honest axes, the right comparison, a title no stronger than the evidence (Day 8).
Is the boundary drawn first?
If a model follows, split before any fit (Day 9).

Why emailed flights load higher Diagram

Seasonpeak or off-peak
- back door: to Load factor
targeting
Emailed70% of peak, 20% of off-peak
true effect +0.5
Load factorpeak about 12 points higher

Marketing aims emails where flights fill anyway, so the raw gap mostly measures the targeting rule.

Precise, and wrong - the memo's estimator, run 150 times Illustration

Inside each season the two dots nearly touch. The big gap is between seasons, and emails go mostly to peak.

The memo’s interval is the narrowest of the three and misses the truth by the widest margin.

Each run’s memo interval is narrow, yet the runs cluster far from the truth: a bigger sample would only tighten the wrong answer. Precision is not accuracy.

Simulated flights with a known +0.5-point email effect, re-implemented from the application script with its own seeded generator, so its numbers differ slightly from the Python run (+8.14, +6.75, +0.30).

The four season-by-email cells, one run with intervals, and the whole study repeated. Bias is where the cloud sits; precision is only how wide it is.

Today's run: two corrections take the 8-point gap to 0.3 Bridge

Memo gap, joined rows8.14 pts
Fix the grainone row per flight: 6.75 +/- 0.57−1.39 pts
Compare within season0.30 +/- 0.51; simulated truth +0.5−6.45 pts
Within season, weighted0.3 pts

Load-factor points, emailed minus not emailed; 3,000 simulated flights; the memo's own interval is +/- 0.44. Grain removes 1.4 points and the fair comparison almost all the rest: neither shows up in the memo's p-value, which only says the biased gap is estimated precisely.

The memo's chart: axis starting at 0.75 Chart

Ratio of the values0.845 / 0.7641.11 x
Ratio of the bars drawn(0.845 - 0.75) / (0.764 - 0.75)6.8 x

Bars encode value by length, so the truncated axis draws an 11% difference as a sevenfold one (Day 8).

The memo and the review Comparison

Criterion	Memo	Senior review
Rows compared	4,855 joined rows, one per email	3,000 flights, one row each
Comparison	All emailed vs all not emailed	Within season, weighted by flights
Uncertainty	p < 0.001 on a biased gap	+0.30 +/- 0.51 points
Title	Email raises load factor 8.1 points	Emailed flights load higher mainly because emails go to peak flights

Today's case is simulated, so the truth is known: marketing emails 70% of peak flights and 20% of off-peak ones, peak flights load about 12 points higher anyway, and the true effect of being emailed is +0.5 points.
The memo stacks three problems: grain (one row per email, fan-out), confounding by season, and a p-value read without a fair interval. Its chart adds a fourth.

Assumptions

Season is the main confounder of email targeting; if marketing also targeted flights with strong early bookings (unrecorded), the within-season comparison is still biased and only a randomized test settles it.
Flights are independent units; flights on the same route and date share demand shocks, so real intervals would be wider than the simple ones here.
The email log is complete and joins on a clean key; Day 6 checks on the log come first in a real review.

Application and workplace decision · 10 min

Run the code. It rebuilds the memo's number from a simulated airline, then applies two corrections in order: one row per flight (grain), then a within-season comparison weighted by flights (confounding), each with a 95% interval. Then (a) say which correction moved the estimate most, (b) state what the final interval rules out, and (c) critique the memo's chart: average load factor as bars on an axis starting at 0.75.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
import pandas as pd

rng = np.random.default_rng(10)
n = 3000  # synthetic flight-days; marketing emails go mostly to peak-season flights
f = pd.DataFrame({"flight_id": range(n), "peak": rng.random(n) < 0.4})
f["emailed"] = rng.random(n) < np.where(f["peak"], 0.7, 0.2)
f["load_factor"] = np.clip(0.74 + 0.12 * f["peak"] + 0.005 * f["emailed"] + rng.normal(0, 0.06, n), 0, 1)
n_emails = np.where(f["emailed"], 1 + rng.poisson(np.where(f["peak"], 2.0, 0.5)), 0)
log = pd.DataFrame({"flight_id": np.repeat(f["flight_id"], n_emails)})  # email log: one row per email sent

def diff(d):  # emailed minus not emailed, in load-factor points, with a 95% CI half-width
    a, b = d.loc[d["emailed"], "load_factor"], d.loc[~d["emailed"], "load_factor"]
    se = np.sqrt(a.var() / len(a) + b.var() / len(b))
    return 100 * (a.mean() - b.mean()), 100 * 1.96 * se

joined = f.merge(log, on="flight_id", how="left")  # memo's query: flights joined to the email log
print(f"rows: flights {len(f)}, joined {len(joined)}")
print("memo (joined rows):    %+.2f pts +/- %.2f" % diff(joined))
print("one row per flight:    %+.2f pts +/- %.2f" % diff(f))
strata = [(diff(g), len(g)) for _, g in f.groupby("peak")]
est = sum(d[0] * m for d, m in strata) / n
half = np.sqrt(sum((d[1] * m / n) ** 2 for d, m in strata))
print(f"within season, weighted: {est:+.2f} pts +/- {half:.2f}  (simulated true effect +0.50)")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 39 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The stratified mean: definition, unbiasedness and its SE derivedadvanced7 minThe estimator behind today's within-season comparison, defined symbol by symbol, its unbiasedness and variance derived line by line, and the t interval for few rows.

The stratified mean: a population mean rebuilt from its strata Formula

\begin{gathered}\mu = \sum_{h=1}^{H} W_h\,\mu_h,\; W_h = N_h / N \\ \bar y_{\text{st}} = \sum_{h=1}^{H} W_h\,\bar y_h \\ \bar y_h = \frac{1}{n_h}\sum_{i \in s_h} y_{hi}\end{gathered}

$h = 1, \dots, H$: the strata: non-overlapping groups that together cover the population (tariff, segment, ward, season)
$N_h,\; N$: units in stratum h and in the whole population; they must be known (or well estimated) from outside the sample
$W_h = N_h / N$: the population share of stratum h: fixed constants, not random, and not the sample share n_h / n
$\mu_h$: the true mean of y inside stratum h; the population mean is exactly the W-weighted average of these
$s_h,\; n_h$: the units sampled in stratum h and how many there are; each stratum is sampled separately
$\bar y_h$: the plain sample mean inside stratum h: an unbiased estimate of mu_h when s_h is a random sample of that stratum
$\bar y_{\text{st}}$: the stratified estimator: each stratum mean is estimated from its own rows, then combined with the population weights

The plain mean of all sampled rows weights stratum h by n_h / n. It equals the stratified mean only when the sample shares match the population shares (proportional allocation); otherwise it averages over a population that does not exist.

Unbiasedness and the variance, derived in six lines Sequence

Linearity
E[ybar_st] = sum W_h E[ybar_h] = sum W_h mu_h = mu: the W_h are constants, each ybar_h unbiased in its stratum.
Independence
Strata are sampled separately: Cov(ybar_h, ybar_k) = 0 for h != k, so the variance of the sum is the sum of variances.
Constants square
Var(W_h ybar_h) = W_h^2 Var(ybar_h): a small stratum contributes little even when it is noisy.
Mean of n_h draws
Var(ybar_h) = (1 - f_h) S_h^2 / n_h, f_h = n_h / N_h; the factor is near 1 for a small sampling fraction.
Plug in
Replace S_h^2 by s_h^2 (divisor n_h - 1, unbiased) and take the root: SE = sqrt(sum W_h^2 s_h^2 / n_h).
Interval
ybar_st +/- 1.96 SE when every n_h is moderate; with few rows per stratum, t with Satterthwaite's df.

The variance of the stratified mean, its plug-in SE and its degrees of freedom Formula

\begin{gathered}\mathrm{Var}(\bar y_{\text{st}}) = \sum_h W_h^2 (1-f_h) \frac{S_h^2}{n_h} \\ \text{SE} = \sqrt{\sum_h a_h}, \quad a_h = \frac{W_h^2 s_h^2}{n_h} \\ \nu \approx \frac{(\sum_h a_h)^2}{\sum_h a_h^2/(n_h-1)}\end{gathered}

$S_h^2$: the true variance of y inside stratum h; only the within-stratum spread enters, never the gaps between stratum means
$s_h^2$: its sample estimate, sum of (y - ybar_h)^2 divided by n_h - 1; needs at least two rows in the stratum
$f_h = n_h / N_h$: the sampling fraction; 1 - f_h is the finite-population correction, dropped when f_h is a few percent or less
$a_h$: stratum h's contribution to the variance; comparing the a_h shows where extra sample would buy the most precision
$\nu$: Satterthwaite's approximate degrees of freedom for the t interval; close to the smallest n_h - 1 when one stratum dominates the variance

The same algebra holds for any weighted sum of independent estimates, which is why today's within-season effect uses it unchanged: replace ybar_h by a stratum's difference in means and s_h^2 / n_h by that difference's variance.

The weights define the question Comparison

Criterion	Population share	Sample share	Volume share
Weight on stratum h	W_h = N_h / N, known from outside the sample	n_h / n, set by whoever designed the sample	share of revenue or consumption
Question it answers	The average per unit: per account, per flight	A population shaped like the sample: none	The average per euro or per kWh
Where you met it	Day 6's MAR fix: post-stratification	Day 7: a fan-out weights rows by their matches	A different estimand: write it next to the number

The plain mean is a weighted mean too. Post-stratification needs no design in advance: re-weighting any sample by known shares (accounts by tariff, patients by ward) gives the stratified estimator, with a slightly larger SE because the n_h are then random.

Worked by hand: ten meters, two strata5 minA utility's average consumption estimated from five homes and five shops: means, variances, the SE and both intervals, every step on paper and checked in Python.

Worked by hand: a utility's average daily consumption from 10 meters Table

Quantity	Residential (h = 1)	Small business (h = 2)	Combined
Accounts N_h, weight W_h	9,000, so W_1 = 0.9	1,000, so W_2 = 0.1	N = 10,000
Sample, kWh per day	8, 10, 12, 9, 11	40, 50, 60, 45, 55	n = 10, five per stratum
Sample mean ybar_h	50 / 5 = 10	250 / 5 = 50	plain mean of all 10 rows: 30
Sample variance s_h^2	(4 + 0 + 4 + 1 + 1) / 4 = 2.5	(100 + 0 + 100 + 25 + 25) / 4 = 62.5	-
W_h x ybar_h	0.9 x 10 = 9.0	0.1 x 50 = 5.0	ybar_st = 14.0
a_h = W_h^2 s_h^2 / n_h	0.81 x 2.5 / 5 = 0.405	0.01 x 62.5 / 5 = 0.125	Var = 0.530
Share of the variance	76%	24%	SE = sqrt(0.530) = 0.73
95% interval, normal	-	-	14.0 +/- 1.96 x 0.73: 12.6 to 15.4
95% interval, t	-	-	Satterthwaite df 6.3, t = 2.42: 12.2 to 15.8

Hypothetical utility, recomputed in Python (numpy, scipy); finite-population factor dropped (each sample under 1% of its stratum). Shops are 10% of accounts but half the sample: 30 is not a noisy 14 but a precise answer about no real population, and more meters on the same 5 / 5 design only narrow it around 30.

From a stratified mean to a stratified effect6 minThe same algebra applied to a difference in means per stratum, ten spend values worked by hand, and the line of today's code that computes it.

From a stratified mean to a stratified effect Formula

\begin{gathered}\hat\tau = \sum_{s} \frac{n_s}{n}\,(\bar y_{s,\text{email}} - \bar y_{s,\text{none}}) \\ \text{SE} = \sqrt{\sum_{s} \left(\tfrac{n_s}{n}\right)^{2} \text{SE}_s^{2}}\end{gathered}

$s$: stratum: peak or off-peak season today; a customer segment, tariff or ward elsewhere
$n_s / n$: share of units in the stratum: the weight, playing the role of W_h in the stratified mean
$\bar y_{s,\text{email}} - \bar y_{s,\text{none}}$: the gap inside one stratum; it estimates that stratum's effect only if assignment inside the stratum is as good as random
$\text{SE}_s^2$: the variance of that gap, s1^2 / n1 + s0^2 / n0: the two arms are disjoint, independent samples, so their variances add (Welch's unpooled form)
$s_{s,1}^2,\; n_{s,1}$: sample variance and count of the treated arm in stratum s (index 0 for the untreated arm)

Comparing within season removes this one measured confounder, not ones nobody recorded. The variances add only while units are disjoint: a customer or flight in both arms or in several strata (repeated weeks) makes the SE too small.

Worked by hand: a retailer's promo email and weekly spend, within segment Table

Quantity	Loyal customers (5)	New customers (5)	Combined
Weekly spend, emailed (EUR)	52, 56, 60	20, 24	plain emailed mean 42.4
Weekly spend, not emailed	50, 54	18, 20, 22	plain not-emailed mean 32.8
Share emailed	3 of 5	2 of 5	plain gap 42.4 - 32.8 = 9.6
Gap d_s	56 - 52 = 4	22 - 20 = 2	-
Var(d_s) = s1^2 / n1 + s0^2 / n0	16 / 3 + 8 / 2 = 9.33	8 / 2 + 4 / 3 = 5.33	-
Weight w_s = n_s / n	0.5	0.5	tau = 0.5 x 4 + 0.5 x 2 = 3.0
Var(tau) = sum w_s^2 Var(d_s)	0.25 x 9.33 = 2.33	0.25 x 5.33 = 1.33	3.67, SE 1.91, 95%: -0.8 to 6.8
Inverse-variance weights instead	0.36	0.64	2.73, SE 1.84: a differently defined average

Hypothetical retailer, verified in Python. Loyal customers spend more anyway and got more emails, so most of the plain gap is segment mix; within segment it is 3.0, interval still including zero. With 2-3 rows per cell the normal interval is a teaching device: real cells need tens of rows, or a t or permutation interval.

The stratified-effect formula, line by line in today's code Table

Formula	Code in the application script
d_s and SE_s, unpooled two-arm variance	se = np.sqrt(a.var() / len(a) + b.var() / len(b)) inside diff(d)
one (d_s, n_s) per season	strata = [(diff(g), len(g)) for _, g in f.groupby('peak')]
tau = sum (n_s / n) d_s	est = sum(d[0] * m for d, m in strata) / n
1.96 SE = sqrt(sum (n_s / n)^2 (1.96 SE_s)^2)	half = np.sqrt(sum((d[1] * m / n) ** 2 for d, m in strata))

diff returns the gap and 1.96 x SE_s, so half is already the 95% half-width: the weights enter squared, exactly as in the formula.

What it estimates: the weighted average of the stratum effects, if and only if assignment inside each stratum is as good as random. That is an assumption about the targeting rule, not something the arithmetic checks.

Properties: why stratifying helps, how to allocate, when it breaksadvanced7 minWithin/between variance decomposition, proportional vs Neyman allocation on the worked utility, a simulation check of the SE, and six failure modes.

Why stratifying pays: the variance it removes Formula

\begin{gathered}S^2 \approx \underbrace{\sum_h W_h S_h^2}_{\text{within}} \\ + \underbrace{\sum_h W_h(\mu_h-\mu)^2}_{\text{between}} \\ V_{\text{SRS}} \approx \frac{S^2}{n} \\ V_{\text{prop}} \approx \frac{\sum_h W_h S_h^2}{n}\end{gathered}

$S^2$: the variance of y over the whole population, split into the spread inside strata and the spread of the stratum means
$V_{\text{SRS}}$: a simple random sample of n units pays for both parts
$V_{\text{prop}}$: proportional allocation (n_h = n W_h) pays only the within part: the between part is removed by design
$V_{\text{Neyman}}$: (sum W_h S_h)^2 / n, reached by the optimal allocation n_h proportional to N_h S_h: sample more where a stratum is large or noisy
$\approx$: exact up to terms of order 1/N and the finite-population factor; the ordering V_Neyman <= V_prop <= V_SRS then holds

Neyman never loses to proportional because (sum W_h S_h)^2 <= sum W_h S_h^2 (Jensen, weights summing to 1). Stratify on variables known before sampling that separate the means (free precision), then allocate toward large and noisy strata; in experiments this is blocked randomization.

The same utility, 10 meters: how the allocation changes the SE Table

Design	Meters: residential / business	Variance of the estimate	SE (kWh per day)
Simple random sample, no strata	whatever mix the draw gives	152.5 / 10 = 15.25	3.91
Proportional, n_h = n W_h	9 / 1	(0.9 x 2.5 + 0.1 x 62.5) / 10 = 0.85	0.92, and one shop leaves no s_h
Equal (the worked sample)	5 / 5	0.405 + 0.125 = 0.53	0.73
Neyman, n_h proportional to N_h S_h	6.4 / 3.6, rounded to 6 / 4	(0.9 x 1.58 + 0.1 x 7.91)^2 / 10 = 0.49	0.70

Treats the worked sample's s_h as the true S_h (illustrative), verified in Python. Of the population variance 152.5, the between-stratum part is 144: a plain random sample pays it, every stratified design removes it. Neyman then shifts meters toward the noisy shop stratum.

The SE formula checked against 400 repeated studies Illustration

Equal samples per stratum over-represent web customers, so the plain mean centres near 50.6. Weighting by population shares centres on 51.6. Spread of the stratified estimates: 1.39; formula SE 1.34; average plug-in SE from each study's own s_h: 1.33.

The formula tracks the observed spread at every size. The noisy web stratum contributes 75% of the variance despite its smaller weight: extra sample there buys the most precision.

Illustrative population with known strata (shares 60/40, means 56/45, SDs 5/13), normal draws from a seeded generator. Every number shown is computed from the simulated studies or from the formula with the true SDs.

Left: equal samples per stratum, two estimators. The plain mean is biased toward the over-sampled stratum; the stratified mean centres on the truth. Right: the formula's SE against the spread actually observed, at five sample sizes.

Six ways the stratified estimate breaks, and the fix for each Overview

A stratum with 0 or 1 rows
No s_h: the SE is undefined or the stratum drops out. Collapse strata before seeing outcomes; plan a minimum n_h.
Weights wrong or estimated
W_h from the sample or an old count. Use known shares; estimated ones (raking) must add their uncertainty.
Strata chosen after the outcomes
The split that makes the result look best. Fix strata from design variables before the analysis.
Rows not independent
One customer in many weeks, one network cell, dates. Use a cluster-robust SE or analyse at cluster level.
Few rows, normal 1.96
Skewed spend or claims: intervals too narrow. Use t with Satterthwaite df, or a within-stratum bootstrap.
Confounder not in the strata
The estimate moves when another recorded variable is added. Randomize (Day 5): strata remove only what they hold.

The first two are arithmetic failures, the last a design failure no formula repairs. With a fixed total, many tiny strata leave each s_h on two or three rows and the degrees of freedom collapse: a few well-filled strata beat many sparse ones; with fixed strata, ybar_st converges to mu as every n_h grows.

The review, check by check3 minEach check as a question with a concrete test, and what it found in today's memo.

The review, as questions with tests Table

Check	Question to ask	Test	Today's finding
Table	Are all the rows there, once?	coverage, keys, integrity, missingness by segment	assumed clean (simulated)
Grain	One row per unit of analysis?	COUNT(*) vs COUNT(DISTINCT flight_id)	3,000 flights became 4,855 rows
Comparison	Why did some units get the treatment?	draw the targeting rule; compare within it	emails follow season
Uncertainty	What range of effects fits the data?	effect with an interval, not a p-value	+0.30 +/- 0.51 points
Chart	Would the encoding change the sentence?	bars from zero; title verb vs design	axis at 0.75, causal title
Boundary	Was anything fit before the split?	fit scopes of every learned step	no model in this memo

Run the rows top to bottom; a failure stops the review, because every later row would be computed on the wrong table.

The grain error2 minHow a join to the email log over-weights peak flights.

How 3,000 flights became 4,855 rows Diagram

Flights3,000 rows
- on flight_id: to Left join
Email logone row per email
1+ per email
Left joinflight repeated per email
4,855 rowspeak flights over-weighted

Peak flights receive more emails, so they appear more often, and they are the highest-loading flights (Day 7).

The within-season comparison in today's case2 minWhich weights today's estimate uses, and why the grain fix has to come first.

Two ways to combine the seasons Comparison

Criterion	By flights (today)	By inverse variance
Weight per season	n_s / n: the season's share of flights	1 / SE_s squared, normalised
Answers	Average effect per flight in this network: the director's question	The most precise average of the season effects
Trade-off	Less precise if one season is small or noisy	A differently defined estimate if effects differ by season

Both need one row per flight first: on the joined table emailed peak flights count once per email, so every n_s and arm mean is wrong. When the season effects agree both weightings target one number; when they differ, report inverse variance only beside the flight-weighted one, labelled as a different estimand.

Reading the final interval2 minWhich sentences the +0.30 +/- 0.51 interval supports.

What the final interval supports Table

Statement	Supported?	Why
Email raises load factor by about 8 points	no	effects above about +0.8 are outside the interval
Email has no effect	no	the interval includes +0.5
Any effect is under one point	yes, if season is the only confounder	upper end about +0.8
The effect is worth the programme's cost	unknown	needs the value of a point and a test

Designing the holdout2 minStrata, size, metrics, unit and report.

The randomized holdout to propose Sequence

Strata
Randomize within peak and within off-peak flights.
Size
Detect the smallest lift that pays for the email programme (Day 4).
Metrics
Load factor and revenue per flight; denied boardings as a guardrail.
Unit
One flight; check spillover between flights on the same route and date.
Report
The effect with its interval, against the cost per point.

What remains uncertainadvanced3 minUnrecorded targeting, shared shocks and an incomplete log - and why only randomization removes the first.

What a within-season comparison cannot fix Illustration

Comparing within season removes the confounder you recorded. The one nobody recorded stays in the estimate, and grows with how hard marketing targets it. A coin flip ignores both.

Simulated flights (2,500 per run, 24 runs per point; bands: middle 80% of runs). The unrecorded signal adds 2 load-factor points per standard deviation; the true email effect stays +0.5.

Add one driver nobody recorded and the adjusted estimate drifts away from the truth as marketing targets it harder. Assignment by coin flip stays on the truth.

What the review still cannot rule out Overview

Unrecorded targeting
Emails sent to flights with strong early bookings stay confounded.
Shared shocks
Flights on one route and date share demand: real intervals are wider.
Incomplete log
A missing email log biases who counts as emailed (Day 6).

Memory anchor

Table, grain, comparison, uncertainty, chart - a precise estimate of a biased comparison is still wrong.

Useful for

Reviewing any analysis or memo before it reaches a decision-maker - the six checks in order, whether it is a claims-cost note in insurance, a churn deck in telecom or a campaign readout in retail.
Estimating a population average from a stratified or re-weighted sample - consumption per meter in energy, readmission rate per ward in a hospital network - with an SE that matches the design.
Self-assessing which Week 2 skill needs studio time before Week 3 modeling.
Practising how to say "the claim is not supported" without saying "there is no effect".

Not suitable for

Exhaustive assessment - a 30-minute checkpoint samples skills; the Sunday studio extends it with a new messy-data variant.
Real causal conclusions about marketing email - the case is simulated to make the true effect known.

Traps

Correcting in the wrong order

Intervals and stratification are computed on a fan-out join or a table with gaps, so every later step is a precise answer to the wrong question.

How to avoid it

Table, grain, comparison, uncertainty, presentation - in that order.

Trading one overclaim for another

"Email adds 8 points" is replaced by "email does nothing", which the interval does not support either.

How to avoid it

State what the evidence rules out and what it still allows, then name the evidence that would decide.

Weighting strata by the sample

A patient survey over-samples small rural clinics on purpose, then reports the plain mean of all responses; or a utility averages equal numbers of home and shop meters. The figure describes the sample design, not the population.

How to avoid it

Combine stratum estimates with population shares W_h, and compute the SE from the within-stratum variances.

Senior reasoning

A senior reviews an analysis in the order its errors compound and quantifies each correction, so the author sees which step mattered. They protect the useful core of a flawed memo - here, the question is worth answering - and turn the disagreement into a test design, with a decision metric and a pre-agreed size of effect that would change the plan. The same review, in the same order, applies to a claims-cost note in insurance or a retention deck in SaaS.

Interview question

"A colleague shows that customers who used feature X spend 30% more. Leadership wants to push feature X to everyone. What do you say?" How would you structure the answer?

Outline your answer first.

Recall cards scheduled after completion

List the six review checks for an analysis, in the order errors compound.
Why can a comparison have p < 0.001 and still be the wrong answer?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

The ASA Statement on p-Values: Context, Process, and Purpose — Ronald L. Wasserstein, Nicole A. Lazar, 2016core
P-values do not measure the probability that a hypothesis is true or the size of an effect, and decisions should not rest on a threshold alone. (Principles 2, 3 and 5)
Fundamentals of Data Visualization — Claus O. Wilke, 2019optional
Proportional ink - bar axes must start at zero - for the chart critique in the case and probe p5. (Chapter 17)
Kimball Dimensional Modeling Techniques: Grain — Kimball Groupoptional
Declaring the grain before combining tables - the first correction in the case.
Common pitfalls and recommended practices (scikit-learn User Guide, section 12) — scikit-learn developersoptional
Fitting preprocessing only on training data, for probe p6. (12.2.2 Data leakage during pre-processing)

Optional sections

Deep dive · 10 min

Why the stratified interval is computed that way. The within-season estimate is a weighted average of two independent differences, $\hat\theta = \sum_s w_s \hat d_s$ with $w_s = n_s / n$ . Its variance is $\sum_s w_s^2 \operatorname{Var}(\hat d_s)$ , so the half-width is $1.96\sqrt{\sum_s w_s^2 SE_s^2}$ . Weighting by flights answers "average effect per flight in this network"; weighting by inverse variance would give a more precise but differently defined estimate. State which one you report.

What the simple interval ignores. Flights on the same date and route share demand shocks, and the same flight appears on many dates. Treating 3,000 flight-days as independent understates the standard error; clustered standard errors or a test randomized at the route-week level would be more honest (Days 15 and 22 return to dependence).

Sizing the test. If the programme costs the equivalent of 0.3 load-factor points per flight, the test must detect roughly that size. With a per-flight standard deviation of about 6 points, the standard error of a two-group difference with $m$ flights per group is $6\sqrt{2/m}$ ; for 80% power at 5% two-sided you need $(1.96 + 0.84) \times SE \le 0.3$ , which gives $m \approx 6{,}300$ flights per group - a sobering number that may change the design (longer test, a stronger metric such as bookings per email, or a decision that the programme is not worth testing).

Week 2 review. Use your probe results: two or more misses in one area (data quality, SQL grain, inference, charts, preprocessing) means that area gets the first block of the Sunday studio.

Interview practice · 10 min

You find a serious flaw in a senior colleague's analysis an hour before it goes to leadership. What do you do?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-11 · depends on numpy, pandas

1 Recall · 5 min

2 Concept · 10 min

Assumptions

3 Application and workplace decision · 10 min

4 Mini-assessment · 8 min

Technical depth · optional · 39 min in total

Useful for

Not suitable for

Traps

Senior reasoning

Interview question

Recall cards scheduled after completion

Prerequisites and related concepts

References

Optional sections

Recall · 5 min

Concept · 10 min

Application and workplace decision · 10 min

Mini-assessment · 8 min