Skip to content
ML Academy

Core day 3 of 60 · Week 1 · Part I · application depth

Probability and Distributions

A fraud screen "catches 95% of fraud with a 2% false-alarm rate". Fraud is about 0.2% of payments, and your team can review 500 alerts a day out of 100,000 payments. How many alerts will be real fraud - and which assumption does that answer rest on?

By the end: You compute a posterior probability with natural frequencies, name the assumption that makes it valid in deployment, and say when a count model (Poisson) understates the risk of extreme values.

At a glance

Count cases instead of multiplying percentages, state the base rate before quoting a precision, and check whether your distribution's tail matches the data.

  • Two directions

    P(alert | fraud) describes the screen; P(fraud | alert) also needs the base rate.

  • Natural frequencies

    Out of 100,000 payments: 190 caught fraud against 1,996 false alarms.

  • Spread

    Variance measures spread; an average of n values varies sigma over root n.

  • Pick the shape

    Binomial, Poisson, normal or heavy-tailed: each assumes a process.

  • Tails

    Same mean, double the variance: extreme no-shows six times as often.

Recall · 5 min

From Day 2: state the grain, cutoff, baseline, metric unit and action cost of the D-14 fare-class decision, and name one column that would leak.

Open due reviews

Sign in to record this block.

Concept · 10 min

Probability is bookkeeping for uncertainty, and most mistakes are bookkeeping mistakes. A screen that flags 95% of fraud sounds as if 95% of its alerts are fraud. They are different numbers: P(alert∣fraud)P(\text{alert} \mid \text{fraud}) describes the screen, P(fraud∣alert)P(\text{fraud} \mid \text{alert}) also depends on how rare fraud is. Counting cases out of 100,000 payments, natural frequencies, makes the difference visible.

With a rare event, even an accurate screen produces mostly false alarms

Predict first

Prevalence 1 in 500, sensitivity 95%, false-alarm rate 2%. Roughly what share of alerts are fraud?

Text description of the figure

A grid of 1,000 squares representing payments. Controls set prevalence, sensitivity and specificity. Fraud cases, caught fraud, missed fraud and false alarms are colored, and the share of alerts that are real fraud is shown as a count fraction.

100,000 payments through the fraud screen Diagram
100,000paymentsfraud rate 0.2%200 fraud99,800 legitimate190 flagged95% of fraud10 missed1,996 flagged2% of legitimate97,804 cleared2,186 alerts190 fraud = 8.7%
  1. 100,000 paymentsfraud rate 0.2%
    • to 99,800 legitimate
  2. 200 fraud
    • to 190 flagged
    • to 10 missed
  3. 99,800 legitimate
    • to 1,996 flagged
    • to 97,804 cleared
  4. 190 flagged95% of fraud
    • to 2,186 alerts
  5. 10 missed
  6. 1,996 flagged2% of legitimate
    • to 2,186 alerts
  7. 97,804 cleared
  8. 2,186 alerts190 fraud = 8.7%

The 95% describes the screen on fraud cases. Fewer than 1 alert in 11 is fraud, because legitimate payments outnumber fraud 499 to 1.

1,000 payments, four worlds: same screen, different meaning of an alert Illustration
Live trafficfraud 0.2%, specificity 98%2 of 22 alerts are fraudprecision 8.7%(exact)
Fraud 1%same screen9 of 29 alerts are fraudprecision 32.4%(exact)
Enriched pilotfraud 5%, same screen47 of 66 alerts are fraudprecision 71.4%(exact)
Better specificityfraud 0.2%, specificity 99.8%2 of 4 alerts are fraudprecision 48.8%(exact)
  • fraud, flagged
  • fraud, missed
  • legitimate, flagged
  • legitimate, cleared

Each grid is 1,000 payments, counts rounded to whole payments; the precision line is the exact value. Sensitivity 95% in every panel. The first three change only the base rate; the fourth changes only the false-alarm rate.

Red cells are false alarms. When fraud is rare they come from the huge legitimate group, so only the false-alarm rate can shrink them.

Bayes' theorem, the same arithmetic Formula
P(F∣A)=P(A∣F) P(F)P(A∣F) P(F)+P(A∣¬F) P(¬F)=0.95×0.0020.95×0.002+0.02×0.998=0.087\begin{aligned} P(F \mid A) &= \frac{P(A \mid F)\,P(F)}{P(A \mid F)\,P(F) + P(A \mid \neg F)\,P(\neg F)} \\ &= \frac{0.95 \times 0.002}{0.95 \times 0.002 + 0.02 \times 0.998} = 0.087 \end{aligned}
P(F)P(F)
prior: the base rate of fraud in the population you score
P(A∣F)P(A \mid F)
sensitivity: share of fraud the screen flags
P(A∣¬F)P(A \mid \neg F)
false-alarm rate: 1 minus specificity
P(F∣A)P(F \mid A)
posterior: share of alerts that are fraud (precision)
Five building blocks and where they come back Overview
  • Random variable

    A numeric outcome; its expected value is what squared-error models estimate (Days 11, 16).

  • Variance

    Spread around the mean; an average of n values has variance sigma squared over n (Days 4, 15).

  • Covariance

    Joint movement; a regression slope is Cov(X, Y) / Var(X) (Days 11, 22).

  • Conditional probability

    P(A | B) with Bayes; the prior is the base rate (Day 13).

  • Distributions

    A model of the process: binomial, Poisson, normal, heavy-tailed.

Same mean, fatter tail, and the buffer that quietly undercovers Illustration
Same mean, different tail10 or morePoissonrate varies (NB)0.00.10.20481216No-shows on one flight (probability)

Both counts average 4 no-shows. Ten or more: 0.8% of flights under Poisson, 4.6% when the rate varies by flight (exact).

Variance vs mean, 40 routes each010202468Mean no-shows per flight on the routevariance = meanmu + mu^2 / 4

The diagnostic: plot each route's variance against its mean. Poisson routes scatter around variance = mean; mixed-rate routes follow mu + mu^2/k and drift further above it as the mean grows.

Share of flights with at most 8Poisson counts97.9%Overdispersed counts92.7%95% target

Where it breaks: a buffer set at the Poisson 95th percentile (8 no-shows) covers 97.9% of Poisson flights but only 92.7% of overdispersed ones; their own 95th percentile is 9.

Left and right: exact probabilities (Poisson mean 4; negative binomial mean 4, k = 4, variance 8). Middle: 40 simulated routes with means 2-8, 150 flights each, per model (seeded).

  • Five building blocks, each reused later: random variable and expected value, Variance and standard error, Covariance, conditional probability with the base rate, and distributions.
  • When counts vary more than a Poisson model allows, they are overdispersed; a statistic's spread over repeated samples is its sampling distribution (Day 4).
  • Details are in the pictures above and in the optional technical depth below.

Assumptions

  • Sensitivity and false-alarm rate measured in a pilot hold in the deployment population; if fraudsters adapt or the case mix differs, the posterior changes.

  • The prevalence used as the prior is the one for the population being scored, not for a review sample enriched with known cases.

  • Poisson and binomial assume independent events with a common rate; dependence or varying rates widen the tails.

Sign in to record this block.

Application and workplace decision · 10 min

Run the code. It:

  1. computes the fraud screen's posterior with natural frequencies;
  2. checks it by simulation;
  3. compares Poisson no-show counts with counts whose rate varies by flight;
  4. compares the spread of a 25-flight average with one flight.

Before running, guess how much more often an overdispersed flight sees 10 or more no-shows than the Poisson model predicts.

Python

import numpy as np
from scipy import stats

# 1) Bayes with natural frequencies: a fraud screen on 100,000 card payments (synthetic rates)
n, prevalence, sensitivity, specificity = 100_000, 0.002, 0.95, 0.98
fraud = n * prevalence
true_alerts = fraud * sensitivity
false_alerts = (n - fraud) * (1 - specificity)
print(f"fraud {fraud:.0f} | true alerts {true_alerts:.0f} | false alerts {false_alerts:.0f}")
print(f"P(fraud | alert) = {true_alerts / (true_alerts + false_alerts):.3f}")

# 2) The same screen simulated, as a check on the arithmetic
rng = np.random.default_rng(3)
is_fraud = rng.random(n) < prevalence
alert = np.where(is_fraud, rng.random(n) < sensitivity, rng.random(n) > specificity)
print(f"simulated P(fraud | alert) = {is_fraud[alert].mean():.3f} from {alert.sum()} alerts")

# 3) No-shows per flight: Poisson implies variance = mean; real counts are often overdispersed
lam = 4.0
poisson = rng.poisson(lam, 20_000)
mixed = rng.poisson(lam * rng.gamma(4.0, 1 / 4.0, 20_000))  # the rate itself varies by flight
for name, x in [("poisson", poisson), ("overdispersed", mixed)]:
    print(f"{name:13s} mean {x.mean():.2f} var {x.var():.2f} P(no-shows >= 10) {np.mean(x >= 10):.4f}")
print(f"Poisson formula P(>= 10) = {stats.poisson.sf(9, lam):.4f}")

# 4) Sampling distribution: a 25-flight average varies far less than one flight
means = rng.poisson(lam, (5_000, 25)).mean(axis=1)
print(f"sd one flight {np.sqrt(lam):.2f} | sd of 25-flight mean {means.std():.3f} (theory {np.sqrt(lam / 25):.3f})")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Sign in to record this block.

Mini-assessment · 8 min

Your first answer is your evidence. You can re-answer later in practice mode; that is labelled practice.

Question 1 · easy

A vendor says its screen "catches 95% of fraud". Which quantity is that statement about?
Options

How confident are you? (optional, used for calibration)

Question 2 · medium

An airline's engine-health monitor raises an alert on 90% of engines that have a developing fault and on 5% of healthy engines. 1% of engines inspected have a developing fault. An alert fires. What is the probability, in percent, that the engine has a developing fault? Use natural frequencies.
percent

How confident are you? (optional, used for calibration)

Question 3 · hard

A fraud model showed 40% precision in a pilot on a review sample where 2% of payments were fraud (known fraud cases were added to make the sample informative). On live traffic, where fraud is 0.2%, operations see about 6% precision. Sensitivity on confirmed live fraud is unchanged at 90%. What is the best explanation?
Options

How confident are you? (optional, used for calibration)

Question 4 · medium

A flight has 180 confirmed bookings. Treat each passenger as independently skipping the flight with probability 0.05 (an illustrative rate). What is the standard deviation of the number of no-shows on that flight? (Two decimals.)
no-shows

How confident are you? (optional, used for calibration)

Question 5 · hard

In today's simulation the average of 25 flights has standard deviation 0.40, while a single flight (Poisson, mean 4.0) has standard deviation 2.00. A planner wants a safety margin for the overbooking limit of one flight and proposes "the mean plus two standard deviations of the 25-flight average", that is 4.0 + 0.8. What is wrong with that margin?
Options

How confident are you? (optional, used for calibration)

Sign in to complete lessons and keep your progress.

Technical depth · optional · 26 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

Definitions first: moments and the spread of an average5 minExpectation, variance, covariance and the standard error defined as math, checked against the Poisson run, and why averages calm down as root n.
The objects, defined once: expectation, variance, standard error Formula
E[Y]=∑yy P(Y=y)Var⁡(Y)=E[Y2]−(E[Y])2se⁡(Yˉ)=σ/n\begin{aligned} \mathbb{E}[Y] &= \textstyle\sum_{y} y\,P(Y{=}y) \\ \operatorname{Var}(Y) &= \mathbb{E}[Y^2] - (\mathbb{E}[Y])^2 \\ \operatorname{se}(\bar{Y}) &= \sigma / \sqrt{n} \end{aligned}
E[Y]\mathbb{E}[Y]
the probability-weighted average of outcomes: the long-run mean, and what squared-error models estimate
Var⁡(Y)\operatorname{Var}(Y)
expected squared distance from the mean; its square root (sd) is back in the units of Y
Cov⁡(X,Y)\operatorname{Cov}(X,Y)
the joint version, E[XY] - E[X]E[Y]; a regression slope is Cov(X,Y)/Var(X) (Day 11)
se⁡(Yˉ)\operatorname{se}(\bar{Y})
the sd of a mean of n INDEPENDENT values; correlated values shrink slower than root n

Check on a count with mean 4 (Poisson): E = 4, Var = 4, and a 25-flight mean has se 2/5 = 0.40, exactly what today's simulation measures.

Five building blocks and where they come back Overview
  • Random variable

    A numeric outcome; its expected value is what squared-error models estimate (Days 11, 16).

  • Variance

    Spread around the mean; an average of n values has variance sigma squared over n (Days 4, 15).

  • Covariance

    Joint movement; a regression slope is Cov(X, Y) / Var(X) (Days 11, 22).

  • Conditional probability

    P(A | B) with Bayes; the prior is the base rate (Day 13).

  • Distributions

    A model of the process: binomial, Poisson, normal, heavy-tailed.

The root-n law is two variance rules applied to a mean Formula
Var⁡(Yˉ)=Var⁡(1n∑i=1nYi)=1n2∑i=1nVar⁡(Yi)=σ2n\operatorname{Var}(\bar Y) = \operatorname{Var}\Big(\tfrac{1}{n}\sum_{i=1}^{n} Y_i\Big) = \tfrac{1}{n^{2}} \sum_{i=1}^{n} \operatorname{Var}(Y_i) = \frac{\sigma^{2}}{n}
Var⁡(aY)=a2Var⁡(Y)\operatorname{Var}(aY) = a^{2}\operatorname{Var}(Y)
scaling rule: the 1/n in front comes out squared
Var⁡(X+Y)\operatorname{Var}(X + Y)
= Var(X) + Var(Y) for INDEPENDENT X and Y: spreads add in variance, never in sd
σ2=4, n=25\sigma^{2} = 4,\ n = 25
one Poisson flight has Var 4; the 25-flight mean has Var 4/25 = 0.16, sd 0.40, the simulated value
σ2n (1+(n−1)ρ)\tfrac{\sigma^{2}}{n}\,(1 + (n-1)\rho)
if every pair is correlated rho: with rho = 0.1 the same mean has sd 0.74, not 0.40

Independence is the load-bearing assumption. Handle times within one call-centre shift, or claims from one storm, are correlated, so their average narrows slower than root n.

The spread of an average shrinks with the square root of n Chart
00.511.5220406080100n = 25: 0.40sigma / root n, sigma = 2 — Flights averaged (n): 1, Standard deviation, no-shows: 2sigma / root n, sigma = 2 — Flights averaged (n): 4, Standard deviation, no-shows: 1sigma / root n, sigma = 2 — Flights averaged (n): 9, Standard deviation, no-shows: 0.667sigma / root n, sigma = 2 — Flights averaged (n): 16, Standard deviation, no-shows: 0.5sigma / root n, sigma = 2 — Flights averaged (n): 25, Standard deviation, no-shows: 0.4sigma / root n, sigma = 2 — Flights averaged (n): 50, Standard deviation, no-shows: 0.283sigma / root n, sigma = 2 — Flights averaged (n): 100, Standard deviation, no-shows: 0.2Flights averaged (n)Standard deviation, no-shows

Exact values for Poisson no-shows with mean 4. Quadrupling n halves the spread; Day 4 builds confidence intervals on this curve.

Bayes derived, then worked by hand5 minThe theorem derived from the definition of conditional probability, the fraud numbers plugged in, and a second posterior worked by hand on a hospital ward.
Bayes derived: one definition, used twice Formula
P(F∣A)=defP(F∩A)P(A)=P(A∣F) P(F)P(A∣F) P(F)+P(A∣¬F) P(¬F)\begin{aligned} P(F \mid A) &\stackrel{\text{def}}{=} \frac{P(F \cap A)}{P(A)} \\ &= \frac{P(A \mid F)\,P(F)}{P(A \mid F)\,P(F) + P(A \mid \neg F)\,P(\neg F)} \end{aligned}
P(F∣A)P(F \mid A)
the definition of conditional probability: the share of A-cases that are also F-cases
P(F∩A)P(F \cap A)
the joint, written the other way round: P(A | F) P(F) - the same definition applied to A given F
P(A)P(A)
law of total probability: alerts come from fraud cases or from legitimate ones, nowhere else

There is no extra axiom in Bayes' theorem: it is the definition of conditional probability read in both directions, with the denominator expanded. The tree computes the same three quantities as counts.

Bayes' theorem, the same arithmetic Formula
P(F∣A)=P(A∣F) P(F)P(A∣F) P(F)+P(A∣¬F) P(¬F)=0.95×0.0020.95×0.002+0.02×0.998=0.087\begin{aligned} P(F \mid A) &= \frac{P(A \mid F)\,P(F)}{P(A \mid F)\,P(F) + P(A \mid \neg F)\,P(\neg F)} \\ &= \frac{0.95 \times 0.002}{0.95 \times 0.002 + 0.02 \times 0.998} = 0.087 \end{aligned}
P(F)P(F)
prior: the base rate of fraud in the population you score
P(A∣F)P(A \mid F)
sensitivity: share of fraud the screen flags
P(A∣¬F)P(A \mid \neg F)
false-alarm rate: 1 minus specificity
P(F∣A)P(F \mid A)
posterior: share of alerts that are fraud (precision)
A posterior by hand: a ward early-warning score (illustrative rates) Sequence
  1. Write the three inputs

    Prevalence 2% of ward patients deteriorate; sensitivity 85%; false-alarm rate 15% of the rest.

  2. Count per 1,000 patients

    20 true cases, 17 of them alert. 980 others, 147 of them alert.

  3. Form the posterior

    P(case | alert) = 17 / (17 + 147) = 17/164 = 10.4%.

  4. Check with the formula

    0.85 x 0.02 / (0.85 x 0.02 + 0.15 x 0.98) = 0.017 / 0.164 = 10.4%. Same number.

  5. Stress the prior

    On a high-risk unit with prevalence 5%, the same screen gives 0.0425 / 0.185 = 23%: the screen did not change.

Verified in Python. One alert in ten is real - the arithmetic behind alarm fatigue, and why the threshold must come from review capacity and costs.

Share of alerts that are fraud, same screen, four base rates Chart
  • Fraud rate 0.2%8.7 %
  • Fraud rate 1%32.4 %
  • Fraud rate 5%what an enriched pilot sample can show71.4 %
  • Fraud rate 20%92.2 %

Sensitivity 95% and specificity 98% in every row; only the base rate changes. Quote a precision only with the population it was measured on.

Bayes as odds, and chaining evidenceadvanced3 minThe odds form, the likelihood ratio an alert needs, and when multiplying two signals double-counts.
Bayes in odds form: multiply, do not divide Formula
P(F∣A)P(¬F∣A)=P(F)P(¬F)×P(A∣F)P(A∣¬F)=2998×0.950.02≈0.095\begin{aligned} \frac{P(F \mid A)}{P(\neg F \mid A)} &= \frac{P(F)}{P(\neg F)} \times \frac{P(A \mid F)}{P(A \mid \neg F)} \\ &= \frac{2}{998} \times \frac{0.95}{0.02} \approx 0.095 \end{aligned}
P(F)P(¬F)\frac{P(F)}{P(\neg F)}
prior odds: 2 fraud per 998 legitimate payments
P(A∣F)P(A∣¬F)\frac{P(A \mid F)}{P(A \mid \neg F)}
likelihood ratio of an alert: 0.95 / 0.02 = 47.5, a property of the screen
0.0950.095
posterior odds; as a probability 0.095 / 1.095 = 8.7%, the same answer as the tree

A second, conditionally independent signal multiplies the odds again by its own likelihood ratio. If the two signals share causes, multiplying double-counts the evidence: two rules that fire on the same device fingerprint are one piece of evidence, not two.

Likelihood ratio an alert needs before it is more likely right than wrong Chart
  • Fraud 0.2% (prior odds 1 to 499)499 x
  • Fraud 1% (1 to 99)99 x
  • Fraud 5% (1 to 19)19 x
  • Today's screen0.95 / 0.0247.5 x

Posterior odds pass 1 only when the likelihood ratio exceeds the inverse of the prior odds: rare events demand extraordinary evidence. The likelihood ratio travels between populations (if the screen's rates do); the posterior never does.

Chaining evidence: multiply likelihood ratios only for independent signals Diagram
Prior odds2 : 998 at a 0.2% fraudratex LR of the alert0.95 / 0.02 = 47.5Independentgiven class?a second signalx LR of signal 2posterior odds multiplyagainModel them jointlyshared cause: samedevice fingerprintyesno
  1. Prior odds2 : 998 at a 0.2% fraud rate
  2. x LR of the alert0.95 / 0.02 = 47.5
  3. Independent given class?a second signal
    • no: to Model them jointly
  4. x LR of signal 2posterior odds multiply again
  5. Model them jointlyshared cause: same device fingerprint

After the first alert the odds are 2/998 x 47.5 = 0.095, a probability of 8.7%. A second signal that shares causes with the first must be scored together with it; multiplying both ratios counts one piece of evidence twice.

Which error rate to improve2 minWhy specificity moves precision far more than sensitivity when the event is rare.
Which improvement helps precision more at a 0.2% base rate? Table
ScreenAlerts per 100,000Share of alerts that are fraud
Sensitivity 95%, specificity 98%2,1868.7%
Sensitivity 99%, specificity 98%2,1949.0%
Sensitivity 95%, specificity 99%1,18816.0%
Sensitivity 95%, specificity 99.8%39048.8%

With rare events, false alarms come from the huge legitimate group, so specificity moves precision far more than sensitivity.

Choosing a distribution, and checking its tail4 minThe process each family assumes, its variance, the variance-to-mean diagnostic and the overdispersion formula.
Four distribution families Comparison
CriterionBinomialPoissonNormalHeavy-tailed
ProcessSuccesses in n independent trials, same probabilityCounts of many independent rare eventsSums and averages of many small effectsA few values dominate the total
Variancen p (1 - p)Equal to the meanA free parameterLarge; the sample mean is unstable
ExampleClaims this year among 500 insured homesDefects per shift; no-shows per flight in today's codeAverage handle time over 100 support ticketsRevenue per customer, insurance claims
Breaks whenTrials are dependent (one storm, many claims) or probabilities differEvents cluster (a bad batch, a group booking); rates differ by unitFew values, heavy tails, hard boundsTreated as normal: risk understated
Overdispersion in one line Formula
Var⁡(Y)=μ+μ2k(Poisson: Var⁡(Y)=μ)\operatorname{Var}(Y) = \mu + \frac{\mu^{2}}{k} \quad (\text{Poisson: } \operatorname{Var}(Y) = \mu)
μ\mu
mean count per flight (4 in today's run)
kk
how much the rate varies by flight; k = 4 gives variance 4 + 16/4 = 8
k→∞k \to \infty
the rate is the same on every flight and the Poisson model returns

A Poisson count whose rate follows a gamma distribution is negative binomial. Today's simulation draws exactly this and measures variance 8.1.

Before trusting Poisson quantiles: the variance-to-mean check Hierarchy
variance / mean ~ 1ratio 2 or morefit k per segmentCounts per segmentroutes, shifts, cells,storesPoisson is plausibleone rate per segmentOverdispersedrates vary; tail muchfatterNegative binomialVar = mu + mu^2 / k
  • Counts per segmentroutes, shifts, cells, stores
    • variance / mean ~ 1Poisson is plausibleone rate per segment
    • ratio 2 or moreOverdispersedrates vary; tail much fatter
      • fit k per segmentNegative binomialVar = mu + mu^2 / k

Dispersion often grows with the mean (variance = mu + mu^2/k), so the busiest segments (large stores, busy cells, busy routes) are where Poisson quantiles fail first. The same check applies to defects per shift (manufacturing) and faults per cell (telecom).

From flags to a review queue3 minTurning a vendor screen into a capacity-limited decision with diminishing returns per slot.
From a vendor's flag to a review queue Sequence
  1. Ask for scores

    A yes/no flag fixes one threshold; a score lets you choose it.

  2. Measure on your traffic

    False-alarm rate on payments like yours, not on the pilot sample.

  3. Rank and cut

    Review the 500 highest scores each day: capacity sets the threshold.

  4. Price the misses

    Compare the cost of a missed fraud with the cost of a review (Day 13).

  5. Monitor the base rate

    If fraud prevalence moves, precision moves with it.

Capacity sets the threshold: a ranked score instead of a vendor flag Illustration
Payments per score bin, one day (log scale)1101001,00010,000vendor cuttop 50099,800 legitimate200 fraud-20246Risk score

The vendor cut flags about 2,186 payments a day, catching 190 of 200 fraud cases. Raising the cut to fill 500 slots still catches about 168: the extra alerts were almost all legitimate.

What a bigger queue buys0%50%100%1001,0002,0003,000Alerts reviewed per day500 slotsvendor cutfraud caught (share of all fraud)precision of the queue

Capacity sets the threshold. Each extra review slot catches less fraud than the one before, while precision keeps falling: price the last slot against the cost of a missed fraud (Day 13).

Computed, not simulated: a binormal score model calibrated to the vendor figures (95% sensitivity and 98% specificity), 100,000 payments a day at 0.2% fraud. Real score distributions are rarely normal; the shape of the trade-off, not the exact counts, is the point.

Library conventions that change the numbersadvanced2 minddof defaults, survival functions, the negative binomial parametrization and integer quantiles.
Conventions that change these numbers (numpy 2.5, pandas 3.0, scipy 1.18) Table
CallConventionTrap
np.var(x), np.std(x)ddof=0: divides by nsmaller than pandas on the same data
Series.var(), Series.std()ddof=1: divides by n - 1mixing the two in one report
stats.poisson.sf(9, mu)P(Y > 9), that is P(Y >= 10)off-by-one when you want 'at least 10'
stats.nbinom(n, p)n = k, p = k / (k + mu) gives mean mu, variance mu + mu^2 / kpassing the mean as p
stats.poisson.ppf(0.95, 4)smallest y with P(Y <= y) >= 0.95: 8a quantile of a count is an integer, coverage is not exactly 95%

Each row was checked against the installed versions. The nbinom row reproduces mean 4, variance 8 and P(Y >= 10) = 4.6%.

Bookkeeping traps2 minFour mistakes that produce confident wrong numbers.
Four bookkeeping traps Overview
  • Flipped conditional

    Reading P(alert | fraud) as P(fraud | alert).

  • Wrong population

    A prior from an enriched sample instead of live traffic.

  • Mean-only fit

    A Poisson model that matches the mean but misses the tail.

  • Hidden dependence

    Claims from one storm, or a group booking, arrive together; independence fails.

Memory anchor

Count people, not percentages: the base rate decides what an alert means, and the tail decides what a count model costs.

Useful for

  • Sizing alert queues (fraud, maintenance, no-show, anomaly alerts) before launch.

  • Explaining to stakeholders why a "95% accurate" screen yields mostly false alarms.

  • Choosing a count model and checking its variance and tail against data before using its quantiles.

Not suitable for

  • Estimating effects of interventions - a conditional probability describes association in the observed population (Day 5).

  • Situations where the rates themselves are unknown and unmeasured in your population; the arithmetic is exact, the inputs are not.

Traps

Quoting pilot precision for production

Pilot samples are often enriched with known positives. Precision and predicted alert volumes then look far better than live traffic delivers.

How to avoid it

Recompute precision with the live base rate, or report sensitivity and false-alarm rate separately and state the population.

Fitting the mean and trusting the tail

A Poisson model matched to the average count (no-shows per flight, claims per storm week, faults per telecom cell) understates how often extreme counts occur when rates vary between units or events cluster.

How to avoid it

Compare the sample variance with the mean and check tail frequencies before using the model's quantiles for decisions.

Senior reasoning

A senior writes the population, the base rate and the conditioning event before any number. They translate rates into counts per day so operations can see the queue, separate properties of the instrument (sensitivity, false-alarm rate) from properties of the population (prevalence), and ask whether the instrument's rates were measured where it will be used. For distributions, they check the variance and tail against data, because decisions such as overbooking and reserves depend on extremes, not on the mean.

Interview question

"A test is 99% accurate and the condition affects 1 in 1,000 people. You test positive. What is the chance you have it?" Walk through it.

Recall cards scheduled after completion

  • Why does an accurate screen for a rare event produce mostly false alarms? Give the natural-frequency calculation.
  • What does overdispersion mean for count data, and what is the standard error of a mean of n independent values?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

References

Optional sections

Deep dive · 10 min

Bayes as odds. Posterior odds = prior odds x likelihood ratio. Fraud prior odds are 2:998; the likelihood ratio of an alert is 0.95 / 0.02 = 47.5; posterior odds are about 0.095, a probability of about 8.7%. The odds form makes chaining independent evidence easy - multiply by each test's likelihood ratio - and it shows why a rare event needs a very high likelihood ratio before an alert becomes more likely right than wrong.

Why the mean, why the median. The constant that minimizes expected squared error is the mean; the constant that minimizes expected absolute error is the median. That is why MSE-trained models estimate conditional means and MAE-trained models conditional medians (Day 16), and why heavy tails pull the two apart.

Heavy tails. In lognormal or Pareto-like data (revenue per customer, insurance claims, outage durations) a few observations carry much of the total. The sample mean converges slowly and jumps when a large value arrives; medians, trimmed means or modeling the log are often more stable, but the business may still need the mean (total revenue). Decide by the decision, then quantify the uncertainty (Day 4 bootstrap).

Overdispersed counts. A Poisson rate that itself varies (here a gamma-distributed multiplier) gives a negative binomial distribution, with variance μ+μ2/k\mu + \mu^2/k. With μ=4\mu = 4 and k=4k = 4 the variance is 8, matching the simulation. Day 26 uses such tails to set overbooking quantiles.

Covariance traps. Covariance measures linear co-movement only: a U-shaped relationship can have zero covariance. Correlation rescales covariance to [-1, 1] but still says nothing about cause (Day 5).

Interview practice · 10 min

An emergency department's model predicts a mean of 4 ambulance arrivals per hour on weekday evenings, and the staffing lead wants enough trolleys and nurses to cover the 95th percentile of arrivals. Which distribution would you use, and how would you check it?

Content version 6 · last verified 2026-10-11 · depends on numpy, scipy