Core day 4 of 60 · Week 1 · Part I · application depth

Statistical Inference

A new booking page lifted conversion by 0.27 percentage points in a two-week test, with p = 0.11. The product manager says "no effect - roll it back". What does the result actually support, and what would you do next?

By the end: You read a confidence interval and a p-value correctly, separate the size of an effect from its statistical significance, and recommend a next step that accounts for power, peeking and multiple comparisons.

At a glance

One test is one draw. Read the effect and its interval in business units, check that the test could detect what matters, and count every look.

Interval first
The interval shows every effect the data allow, in business units.
p-value
How surprising the data are if nothing changed; not the chance the page works.
Power
At 20,000 visitors per arm, a real 0.30 pp lift is detected only 41% of the time.
Many looks
Twenty peeks at an A/A test produce a false win 24% of the time.
Significant is not big
With millions of visitors, a lift too small to pay becomes significant.

Recall · 5 min

From Day 3: what is the standard error of a mean of n independent values, and why did the 25-flight average vary five times less than a single flight?

Open due reviews

Concept · 10 min

You run the experiment once, but inference reasons about running it many times. The lift you measured is one draw from a sampling distribution. A confidence interval and a p-value describe that procedure; neither gives the probability that the page works, and neither says whether the effect is worth money.

Run the same experiment many times - intervals miss 5% of the time and p-values are uniform when nothing changed

Predict first

In 1,000 A/A tests (no true difference), roughly how many give p < 0.05?

Text description of the figure

A simulation that repeats an experiment with a chosen true effect, standard deviation and sample size. One panel stacks the confidence intervals, colored by whether they cover the true value; another shows the histogram of p-values; a third shows how a fixed tiny effect becomes significant as n grows.

Four ideas about one experiment Overview

Estimate
Lift 0.27 pp: one draw from the sampling distribution.
Interval
-0.07 to +0.62 pp: a procedure that covers the truth 95% of the time.
p-value
0.113: how often a gap at least this large appears if nothing changed.
Power
0.41: the chance this design detects a true 0.30 pp lift.

Fifty runs of the same test: the procedure is right 95% of the time Illustration

48 of 50 intervals cover the true lift; 21 of 50 tests are significant. Typical half-width 0.34 pp.

48 of 50 intervals cover the true lift; 40 of 50 tests are significant. Typical half-width 0.21 pp.

interval covers the true lift
interval misses it
filled dot: significant at 5%; hollow: not significant

Simulated: 50 repeats per panel of a two-arm test, control 3.0%, new page 3.3%; conversions drawn from the normal approximation to the binomial. The seed was picked as a typical draw: coverage and significance near their expected 95% and power (0.41, 0.80). Same interval and z-test as the Python block.

Coverage is a property of the procedure across repetitions. Significance (filled dots) is rare at today's size and common at the planned size, with the same true lift.

Reading a test result Diagram

Effect and intervalin business units
- to Excludes zero?
Rerunsized for the lift that pays
Excludes zero?
- yes: to Lower end pays?
no
Powered for value?
- no: back to Rerun
- yes: to Rule it out
Lower end pays?
- yes: to Ship
- no: to Value call
Rule it outthe paying lift is excluded
Shipwith guardrails
Value callreal but maybe too small

"Not significant" sends you to power, not to "no effect". A well-powered null rules out the lifts that would pay.

Under no difference p-values are flat; under a real lift they pile up near zero Illustration

Flat: every 0.05-wide bin holds about 5%. 5.1% fall below 0.05, all of them false wins.

41% below 0.05: that share is the power of this design.

80% below 0.05: that share is the power of this design.

Simulated: 2,000 two-arm tests per panel, control 3.0%, conversions from the normal approximation to the binomial, pooled two-sided z-test (seeded). Dashed line: the 5% per bin a flat histogram would give.

Power to detect a 0.30 pp lift (3.0% to 3.3%) Chart

Normal-approximation power, computed exactly for each sample size (the simulation measured 0.41 at 20,000).

Read a test in this order: effect in business units, its interval, the value threshold, then power.
Key terms: Type I and Type II errors, multiple comparisons and the Bayesian credible interval.
All of it assumes independent units; autocorrelated daily values make intervals too narrow (Day 22).

Assumptions

Units (visitors) are randomized and independent, and each is counted in one arm only; repeat visitors and shared devices weaken this.
The sample size and analysis were fixed in advance; stopping when the result looks good breaks the stated error rates.
The normal approximation holds - fine for thousands of visitors and a few percent conversion, poor for tiny counts or heavy-tailed revenue.

Application and workplace decision · 10 min

Run the simulation of the booking-page test (synthetic: control 3.0%, new page truly 3.3%, 20,000 visitors per arm). It shows:

one test as you would see it;
power and coverage across 4,000 repeats, and A/A tests;
20 peeks at an A/A test;
a tiny lift with 2 million visitors per arm.

Before running, guess the power of the 20,000-visitor test.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
from scipy import stats

rng = np.random.default_rng(3)
def ztest(x_a, x_b, n_a, n_b):
    """Difference in conversion (B - A), 95% CI and two-sided p-value (normal approximation)."""
    p_a, p_b = x_a / n_a, x_b / n_b
    diff = p_b - p_a
    se = np.sqrt(p_a * (1 - p_a) / n_a + p_b * (1 - p_b) / n_b)
    pooled = (x_a + x_b) / (n_a + n_b)
    z = diff / np.sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
    return diff * 100, (diff - 1.96 * se) * 100, (diff + 1.96 * se) * 100, 2 * stats.norm.sf(np.abs(z))

# 1) One booking-page test: control converts at 3.0%, the new page truly at 3.3% (synthetic)
n, reps = 20_000, 4_000
d, lo, hi, p = ztest(rng.binomial(n, 0.030), rng.binomial(n, 0.033), n, n)
print(f"one test: lift {d:.2f} pp, 95% CI [{lo:.2f}, {hi:.2f}] pp, p = {p:.3f}")

# 2) The same test repeated: power, CI coverage, and A/A tests where nothing changed
d, lo, hi, p = ztest(rng.binomial(n, 0.030, reps), rng.binomial(n, 0.033, reps), n, n)
print(f"power: {np.mean(p < 0.05):.2f} | CI covered the true 0.30 pp: {np.mean((lo < 0.3) & (hi > 0.3)):.3f}")
d, lo, hi, p = ztest(rng.binomial(n, 0.030, reps), rng.binomial(n, 0.030, reps), n, n)
print(f"A/A: share p < 0.05 = {np.mean(p < 0.05):.3f} | share p < 0.50 = {np.mean(p < 0.5):.3f}")

# 3) Peeking: A/A tests checked after every 1,000 visitors per arm, 20 looks
looks = np.arange(1, 21) * 1_000
xa = rng.binomial(1_000, 0.030, (reps, 20)).cumsum(axis=1)
xb = rng.binomial(1_000, 0.030, (reps, 20)).cumsum(axis=1)
p_looks = ztest(xa, xb, looks, looks)[3]
print(f"A/A with 20 peeks: share ever 'significant' = {np.mean((p_looks < 0.05).any(axis=1)):.3f}")

# 4) A tiny true lift (3.00% -> 3.05%) with 2 million visitors per arm
N = 2_000_000
d, lo, hi, p = ztest(rng.binomial(N, 0.0300, reps), rng.binomial(N, 0.0305, reps), N, N)
print(f"huge n: power {np.mean(p < 0.05):.2f} for a 0.05 pp lift; typical CI half-width {np.mean(hi - lo) / 2:.3f} pp")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 32 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The sampling distribution, and why it is normal4 minThe CLT stated precisely, what it buys (one reference distribution for every average), and the three places its conditions fail.

The central limit theorem, which makes all of today work Formula

\begin{aligned} &X_1, \dots, X_n \text{ i.i.d.},\ \mathbb{E}[X_i] = \mu,\ \operatorname{Var}(X_i) = \sigma^2 \\ &\Rightarrow\; \frac{\bar{X} - \mu}{\sigma / \sqrt{n}} \xrightarrow{d} \mathcal{N}(0, 1) \end{aligned}

$\text{i.i.d.}$: independent, identically distributed: the assumption that fails first in practice (repeat visitors, daily KPIs)
$\bar{X}$: any average - including a conversion rate, which is an average of 0/1 outcomes
$\xrightarrow{d}$: the distribution of the standardized mean approaches normal, whatever shape each X_i has
$\sigma/\sqrt{n}$: the standard error: the sd of the sampling distribution, from Day 3's variance algebra

The object under study is the distribution of the STATISTIC over imagined repeats, not the data's histogram: normal with sd sigma / root n whatever the data's shape. For a conversion rate, p-hat is approximately N(p, p(1-p)/n). The tree below shows where that promise fails.

Three places the normal approximation breaks Hierarchy

z-test valid here?check the metric and the units
- repeat visitorsDependent unitseffective n below rows (Day 22)
- revenue per userHeavy tailsbootstrap or winsorize, in the plan
- rare conversionsTiny countsnp below 10: exact or simulated test
- none of theseCLT holdsz-test and 1.96 intervals

Dependence makes real intervals wider than computed ones; heavy tails make the approximation arrive slowly; tiny counts make it arrive never. Each fix is chosen before the data arrive, not after the first p-value.

The z-test derived, and the interval as its inverse8 minThe test statistic built from the CLT, both tests worked by hand (today's z-test, a five-value Welch t-test), and the interval as the set of nulls the test keeps.

The z-test derived: standardize the gap, then ask how surprising it is Formula

\begin{aligned} z &= \frac{(\hat{p}_B - \hat{p}_A) - 0}{\sqrt{\hat{p}(1-\hat{p})\left(\tfrac{1}{n_A} + \tfrac{1}{n_B}\right)}} \overset{H_0}{\approx} \mathcal{N}(0,1) \\ p\text{-value} &= 2\,\Phi(-|z|) \end{aligned}

$0$: the null hypothesis: both pages share one conversion rate, so the expected gap is zero
$\hat{p}$: the pooled rate, used because the se is computed under H0, where both arms share it
$z$: how many standard errors the observed gap is from what H0 expects - the CLT makes this N(0,1)
$2\,\Phi(-|z|)$: two-sided tail area: the probability, under H0, of a gap at least this extreme
$t_{n-1}$: when sigma must be estimated from few units (a t-test), the statistic has fatter tails; with thousands of visitors t and z agree

Every classical test is this recipe: a statistic whose distribution is known when nothing is going on, and the tail area of what you observed. Today's run computes z = 1.58, p = 0.113.

The interval for a difference in conversion Formula

\hat d \pm 1.96\sqrt{\frac{\hat p_A(1-\hat p_A)}{n_A} + \frac{\hat p_B(1-\hat p_B)}{n_B}}

$\hat d$: observed lift, p_B minus p_A
$n_A, n_B$: visitors per arm; quadrupling them halves the width
$1.96$: the normal quantile for a 95% interval

At 3.0% and 3.3% with 20,000 per arm the half-width is about 0.34 pp, larger than the lift being tested.

The interval is the test inverted Formula

\begin{aligned} &P\!\left(-1.96 \le \frac{\hat{d} - d}{\mathrm{se}} \le 1.96\right) = 0.95 \\ &\Rightarrow\; d \in \hat{d} \pm 1.96\,\mathrm{se} \end{aligned}

$\hat{d} - d$: estimate minus the true lift; the CLT pins down its distribution before the data arrive
$\Rightarrow$: rearrange the two inequalities for d: the random thing is the interval, not the true lift
$\hat{d} \pm 1.96\,\mathrm{se}$: equivalently: every null value d0 that the z-test would NOT reject at 5%

That is why '95%' belongs to the procedure: before sampling, the interval has a 95% chance of landing on the truth. After sampling it either did or did not - which is also why p = 0.11 and a CI crossing zero are the same statement.

Two ways to be wrong Map

Type I error

A false win. Rate alpha, often 5%, if you look once.

Correct detection

Probability = power. Rises with n and effect size.

Correct

Probability 1 - alpha.

Type II error

A missed lift. Probability 1 - power: 59% in today's test.

Worked by hand: today's booking-page test through the z-test Sequence

Counts
Control 594 of 20,000 converted (2.970%); new page 649 of 20,000 (3.245%). Gap d = 0.275 pp.
Pooled rate under H0
(594 + 649) / 40,000 = 0.0311: if nothing changed, both arms share this rate.
Standard error under H0
sqrt(0.0311 x 0.9689 x (1/20,000 + 1/20,000)) = 0.00174, that is 0.174 pp.
Standardize
z = 0.275 / 0.174 = 1.58 standard errors from what H0 expects.
Tail area
p = 2 x Phi(-1.58) = 0.113. The unpooled se (0.00174 here too) gives the interval 0.275 +/- 1.96 x 0.174 = [-0.07, +0.62] pp.

The counts are the seeded draw behind the application's first printout; every line was recomputed in Python and matches it. Redo it with a calculator once - it is the whole test.

Worked by hand: Welch's t-test on five lab turnaround times per workflow (illustrative) Sequence

Data, minutes
Old routing: 42, 38, 45, 40, 35 (mean 40.0). New routing: 33, 36, 30, 35, 31 (mean 33.0). Gap 7.0 minutes.
Sample variances (n - 1)
Old: (4 + 4 + 25 + 0 + 25) / 4 = 14.5. New: (0 + 9 + 9 + 4 + 4) / 4 = 6.5.
Standard error
sqrt(14.5/5 + 6.5/5) = sqrt(2.9 + 1.3) = sqrt(4.2) = 2.05 minutes; no pooling of variances.
Statistic
t = 7.0 / 2.05 = 3.42.
Welch degrees of freedom
4.2^2 / (2.9^2/4 + 1.3^2/4) = 17.64 / 2.525 = 6.99, so about 7 df instead of 8.
p-value and interval
Two-sided p = 0.011 from t with 6.99 df; 95% CI 7.0 +/- 2.37 x 2.05 = [2.2, 11.8] minutes.
Why t, not z
A normal reference would give p = 0.0006: with 5 values per group the estimated variance is itself noisy, and t's fatter tails price that in.

Verified with scipy.stats.ttest_ind(old, new, equal_var=False), which returns t = 3.42, df = 6.99, p = 0.011. Healthcare lab data; the same recipe compares machine cycle times or delivery durations.

z-test or Welch's t-test? Comparison

Criterion	z-test, two proportions	Welch's t-test, two means
Data	conversion counts, thousands of visitors	measurements, small groups
Variance	pooled rate under H0	estimated per group, never pooled
Reference distribution	N(0, 1)	t with Welch df (6.99 in the lab example)
Typical use	A/B tests of conversion	lab turnaround, machine cycle times

Both are (estimate - null) / se. With an estimated variance the statistic follows t, whose fatter tails matter below a few dozen units per group and vanish at thousands of visitors.

Power algebra, solved for the sample size5 minPower derived in one line, the sample-size formula that drops out, and a worked sizing of a SaaS activation test, verified by simulation.

Power algebra: solve for the sample size Formula

\begin{aligned} 1 - \beta &= \Phi\!\left(\frac{\delta}{\mathrm{se}(\hat{d})} - z_{1-\alpha/2}\right) \\ \Rightarrow\; \frac{\delta}{\mathrm{se}} &= z_{1-\alpha/2} + z_{1-\beta} \end{aligned}

$\delta$: the true lift you want to detect; under it, z is centred at delta/se instead of 0
$\Phi(\cdot)$: power = chance the shifted statistic still clears the 1.96 cutoff
$z_{1-\beta}$: 0.84 for 80% power: the gap must be 1.96 + 0.84 = 2.8 standard errors wide
$\mathrm{se}^2 = \tfrac{p_A(1-p_A) + p_B(1-p_B)}{n}$: substitute and solve for n: the sample-size formula below drops out

One line to remember: a test detects reliably only effects about 2.8 standard errors wide. Check it on today's test: se = 0.00175 at 20,000 per arm, delta / se = 1.72, power = Phi(1.72 - 1.96) = 0.40; the simulation measured 0.41.

Visitors per arm for a target power Formula

n = \frac{(z_{1-\alpha/2} + z_{1-\beta})^{2}\,[p_A(1-p_A) + p_B(1-p_B)]}{(p_B - p_A)^{2}}

$z_{1-\alpha/2}$: 1.96 for a two-sided 5% test
$z_{1-\beta}$: 0.84 for 80% power
$p_B - p_A$: the smallest lift worth detecting: 0.30 pp gives about 53,000 per arm

The effect size enters squared, so halving the lift you want to detect quadruples the sample. That is why 'detect any change' is not a design: the denominator goes to zero.

Worked by hand: sizing a SaaS activation test (illustrative numbers) Sequence

Fix the decision inputs
Onboarding flow converts 10% of sign-ups; the smallest lift that pays is +1 pp, so p_B = 11%. Alpha 5%, power 80%.
Variance term
p_A(1-p_A) + p_B(1-p_B) = 0.090 + 0.098 = 0.188.
Plug into the formula
n = (1.96 + 0.84)^2 x 0.188 / 0.01^2 = 7.84 x 0.188 / 0.0001.
Solve
About 14,750 sign-ups per arm, 29,500 in total. Verified by simulation: 80% of such tests come back significant.
Sanity-check the duration
At 1,000 sign-ups a day the test runs a month. Too slow? Raise the minimum worthwhile lift or reduce variance - not the power.

The same five lines size a retention email (marketing) or a triage rule pilot (healthcare); only the rates and the worthwhile lift change.

Power to detect a 0.30 pp lift (3.0% to 3.3%) Chart

Normal-approximation power, computed exactly for each sample size (the simulation measured 0.41 at 20,000).

Buying power without more trafficadvanced3 minVariance reduction with a pre-period covariate, and what it can and cannot do.

Variance reduction with a pre-period covariate (CUPED-style adjustment) Formula

\begin{aligned} Y^{\mathrm{adj}}_i &= Y_i - \theta\,(X_i - \bar X), \quad \theta = \frac{\operatorname{Cov}(X, Y)}{\operatorname{Var}(X)} \\ \operatorname{Var}(\bar Y^{\mathrm{adj}}) &= (1 - \rho^{2})\operatorname{Var}(\bar Y) \end{aligned}

$X_i$: the same metric for visitor i before the test started; unaffected by the treatment
$\theta$: regression slope of Y on X, estimated on both arms pooled
$\rho$: correlation between pre-period and in-test values
$1 - \rho^{2}$: variance factor: rho = 0.5 removes a quarter of the variance, so the same power needs a quarter fewer visitors

The lift estimate stays unbiased because X is measured before randomization. It helps only where a pre-period value exists and correlates (returning visitors, revenue per user).

Visitors needed after a pre-period adjustment (% of unadjusted) Chart

rho = 0 (no covariate)100 %
rho = 0.391 %
rho = 0.575 %
rho = 0.7about half the visitors51 %
rho = 0.919 %

Required sample scales with variance, so it shrinks by 1 - rho^2. The covariate must be measured before assignment: adjusting for anything the treatment can move biases the lift.

Many looks, many segments4 minHow false wins accumulate over looks and segments, and what a planned boundary restores.

Every look is another chance for noise to cross the line Illustration

12 of these 40 A/A tests cross the 5% line at some look (red) and would be shipped as wins if the team stopped there. At the final look alone only 3 are beyond it.

Stopping at the first p < 0.05 inflates false wins with every extra look. A boundary fixed in advance for 20 looks (here |z| > 2.69, calibrated in this simulation) keeps them at 5%: that is what a sequential design buys.

Simulated: 2,000 A/A tests (3.0% conversion in both arms), 1,000 visitors per arm per batch, normal approximation per batch, pooled z-statistic (seeded). Looks are spread evenly over the 20 batches.

Chance of at least one false win among independent tests Chart

1 test5 %
5 tests22.6 %
10 tests40.1 %
30 testssegments, metrics or variants78.5 %

1 - 0.95^m for m independent tests at alpha 0.05. The same arithmetic governs ad creatives (media), store layouts (retail) or control charts on one production line: whoever scans many results owns it. Correct (Bonferroni: alpha / m), pre-register one primary metric, or use a sequential design.

Today's run: false wins in A/A tests Chart

One look at the endnominal 5%5.2 %
20 looks, stop at first p < 0.0524.3 %

4,000 simulated A/A tests (no true difference) per row. Peeking is multiple testing in time: the looks share data, so they inflate less than the same number of independent tests would, but every extra look adds false wins.

Looking early: three designs and what each costs Comparison

Criterion	Unplanned peeking	Group-sequential	Always-valid
Looks	whenever the dashboard is open	a fixed number, planned	continuous monitoring
Boundary	the same 0.05 at every look	stricter, from an alpha-spending plan	wider, valid at any stopping time
False-win rate	grows with every look (24% at 20)	held at the planned 5%	held at 5% whenever you stop
Price	paid later, in false wins	a slightly larger maximum sample	less power than a fixed horizon

Group-sequential designs fix the number of looks and a stricter boundary in advance (alpha spending); always-valid methods allow continuous monitoring at a power cost. Both are chosen before launch.

Trust checks and library defaultsadvanced3 minSample-ratio, A/A and unit checks, and the defaults in scipy and statsmodels that silently change a result.

Checks before reading any result Sequence

Sample ratio
Assignment counts match the planned split; a chi-square test on counts flags broken randomization.
A/A or pre-period
No difference before treatment, or in an A/A run of the same pipeline.
Unit of analysis
Randomized by visitor, analysed by visitor; not by session or by day.
Guardrails
Latency, errors and revenue did not move in the wrong direction.
Planned analysis
The test, the horizon and any corrections are the ones written down before launch.

A sample-ratio mismatch invalidates the comparison however small the p-value; see Kohavi et al. for the trust checks used in online experiments.

Defaults that change a test result (scipy 1.18, statsmodels 0.15) Table

Call	Default	Consequence
scipy.stats.ttest_ind	equal_var=True	Student's t, not Welch: pass equal_var=False when variances or sizes differ
statsmodels proportions_ztest	prop_var=False	standard error from the pooled proportion, as in today's z-test
statsmodels multipletests	method='hs'	Holm-Sidak (family-wise), not Benjamini-Hochberg; name the method explicitly
scipy.stats.false_discovery_control	method='bh'	returns Benjamini-Hochberg adjusted p-values, controlling the false-discovery rate
scipy.stats.bootstrap	method='BCa', n_resamples=9999	bias-corrected intervals; still assumes independent units

Each default was read from the installed library signatures. Reports should state the test, the variance assumption and the correction used.

The Bayesian viewadvanced2 minWhat a credible interval states and what it costs.

Confidence and credible intervals Comparison

Criterion	Frequentist 95% CI	Bayesian 95% credible interval
What it states	The procedure covers the true lift in 95% of repetitions	The lift lies in this range with 95% probability, given the prior
Needs	A fixed design and analysis plan	A prior, stated and defended
Main risk	Read as a probability about this one lift	A convenient prior that decides the answer

With weak priors and large samples the two are often numerically close; the sentences they support differ.

Designing the rerun, and the misreadings to avoid3 minFive decisions to write down before the first visitor, and the four sentences that turn a correct test into a wrong decision.

Before the test starts Sequence

One primary metric
Conversion per visitor; revenue per visitor and page latency as guardrails.
Smallest lift that pays
Set from money, not from what looks achievable.
Sample size
Fix n for 80% power at that lift; write it down.
Analysis plan
Test, interval and segments decided before data arrive.
Looks
One look at the end, or a sequential design with planned looks.

Four misreadings Overview

p as probability
p = 0.11 is not an 11% chance that the null is true.
Null as proof
Not significant with low power says almost nothing.
Significant as large
With huge n, tiny lifts become significant.
Days as units
Autocorrelated daily values give intervals that are too narrow.

Memory anchor

Read the interval in money, check the power, count the looks - the p-value comes last.

Useful for

Reading A/B test results in product, pricing-page, email and app experiments.
Sizing experiments before they start (minimum detectable effect, power, duration).
Challenging segment-mined or dashboard "wins" before they reach a launch decision.

Not suitable for

Observational comparisons without randomization - a small p-value does not remove confounding (Day 5).
Time series of daily KPIs treated as independent draws; autocorrelation invalidates the standard errors (Day 22).

Traps

Calling a non-significant result "no effect"

An underpowered test misses a valuable effect and the team abandons a good change, or claims a risky change is "safe" because nothing was significant.

How to avoid it

Quote the interval against the business threshold and report the power for the effect that matters.

Peeking and segment mining

Repeated looks and many subgroups make at least one false "significant" result likely; it gets shipped and the lift never appears.

How to avoid it

Fix the sample size, the primary metric and the segments in advance; use sequential methods or corrections when you must look more.

Senior reasoning

A senior sizes the test before it starts, from the smallest effect worth acting on, and writes the analysis plan down. They report effect sizes with intervals in business units, treat p-values as one input among several, and separate exploratory findings from confirmatory ones. They know that the stated error rates belong to the procedure and hold only if the procedure was followed.

Interview question

"Your A/B test came back with p = 0.08. The PM wants to know if the feature works." What do you say?

Outline your answer first.

Recall cards scheduled after completion

State correctly what a 95% confidence interval and a p-value of 0.03 mean.
Why can a valuable effect be non-significant, a negligible one significant, and a null effect "win"?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

The ASA Statement on p-Values: Context, Process, and Purpose — Ronald L. Wasserstein, Nicole A. Lazar, 2016core
The ASA's six principles - p-values do not give the probability that a hypothesis is true, do not measure effect size or importance, and decisions should not rest on a threshold alone. (The six principles)
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations — Sander Greenland, Stephen J. Senn, Kenneth J. Rothman, John B. Carlin, Charles Poole, Steven N. Goodman, Douglas G. Altman, 2016optional
Catalogue of common misreadings, including the p-value as the probability of the null (#1), non-significance as no effect (#6), significance as importance (#7) and a specific interval having a 95% chance of containing the truth (#19). (Misinterpretations 1, 6, 7 and 19)
Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing — Ron Kohavi, Diane Tang, Ya Xu, 2020optional
Practical treatment of the statistics behind online experiments - power, variance and pitfalls in A/B tests. (Ch. 17 The Statistics behind Online Controlled Experiments; ch. 18 Variance Estimation and Improved Sensitivity)

Optional sections

Deep dive · 10 min

Sample size from the decision. For two proportions, the per-arm sample size for power $1-\beta$ at two-sided level $\alpha$ is roughly $n = (z_{1-\alpha/2} + z_{1-\beta})^2 \,[p_A(1-p_A) + p_B(1-p_B)] / \delta^2$ . With 3.0% vs 3.3%, $\alpha = 0.05$ and 80% power: $(1.96 + 0.84)^2 \times 0.0610 / 0.003^2 \approx 53{,}000$ per arm. Halving the minimum detectable effect quadruples $n$ - which is why "detect any change" is not a design.

Bootstrap intervals. For metrics without a convenient formula (revenue per visitor with heavy tails, ratios), resample visitors with replacement many times and take percentiles of the recomputed statistic. It still assumes independent units, and it struggles with very heavy tails and small samples.

Corrections. Bonferroni controls the chance of any false positive by testing each of $m$ hypotheses at $\alpha/m$ ; it is simple and conservative. Benjamini-Hochberg controls the expected share of false discoveries and suits screening many metrics. Neither fixes a hypothesis chosen after looking at the data; confirmation does.

Bayesian and frequentist in practice. Frequentist procedures guarantee long-run error rates; Bayesian analysis answers "how probable is a lift above +0.2 pp given the data and the prior". Both need a plan made before the data arrive, and both break under optional stopping if the model ignores it.

Why independence fails for time series. Today's bookings resemble yesterday's. Positive autocorrelation means 60 daily values carry less information than 60 independent ones, so naive standard errors are too small and p-values too optimistic. Day 22 measures this with the ACF; Day 5 uses designs that respect time.

Interview practice · 10 min

Design the analysis plan for an A/B test of a new checkout flow for an online grocery retailer before it launches.

Answer as you would in the interview.

Content version 6 · last verified 2026-10-11 · depends on numpy, scipy

1 Recall · 5 min

2 Concept · 10 min