Probability and Distributions
By the end: You compute a posterior probability with natural frequencies, name the assumption that makes it valid in deployment, and say when a count model (Poisson) understates the risk of extreme values.
At a glance
Count cases instead of multiplying percentages, state the base rate before quoting a precision, and check whether your distribution's tail matches the data.
Two directions
P(alert | fraud) describes the screen; P(fraud | alert) also needs the base rate.
Natural frequencies
Out of 100,000 payments: 190 caught fraud against 1,996 false alarms.
Spread
Variance measures spread; an average of n values varies sigma over root n.
Pick the shape
Binomial, Poisson, normal or heavy-tailed: each assumes a process.
Tails
Same mean, double the variance: extreme no-shows six times as often.
Recall
From Day 2: state the grain, cutoff, baseline, metric unit and action cost of the D-14 fare-class decision, and name one column that would leak.
Concept
Probability is bookkeeping for uncertainty, and most mistakes are bookkeeping mistakes. A screen that flags 95% of fraud sounds as if 95% of its alerts are fraud. They are different numbers: describes the screen, also depends on how rare fraud is. Counting cases out of 100,000 payments, natural frequencies, makes the difference visible.
Predict first
Prevalence 1 in 500, sensitivity 95%, false-alarm rate 2%. Roughly what share of alerts are fraud?
Text description of the figure
A grid of 1,000 squares representing payments. Controls set prevalence, sensitivity and specificity. Fraud cases, caught fraud, missed fraud and false alarms are colored, and the share of alerts that are real fraud is shown as a count fraction.
- 100,000 paymentsfraud rate 0.2%
- to 99,800 legitimate
- 200 fraud
- to 190 flagged
- to 10 missed
- 99,800 legitimate
- to 1,996 flagged
- to 97,804 cleared
- 190 flagged95% of fraud
- to 2,186 alerts
- 10 missed
- 1,996 flagged2% of legitimate
- to 2,186 alerts
- 97,804 cleared
- 2,186 alerts190 fraud = 8.7%
The 95% describes the screen on fraud cases. Fewer than 1 alert in 11 is fraud, because legitimate payments outnumber fraud 499 to 1.
- fraud, flagged
- fraud, missed
- legitimate, flagged
- legitimate, cleared
Each grid is 1,000 payments, counts rounded to whole payments; the precision line is the exact value. Sensitivity 95% in every panel. The first three change only the base rate; the fourth changes only the false-alarm rate.
Red cells are false alarms. When fraud is rare they come from the huge legitimate group, so only the false-alarm rate can shrink them.
- prior: the base rate of fraud in the population you score
- sensitivity: share of fraud the screen flags
- false-alarm rate: 1 minus specificity
- posterior: share of alerts that are fraud (precision)
Random variable
A numeric outcome; its expected value is what squared-error models estimate (Days 11, 16).
Variance
Spread around the mean; an average of n values has variance sigma squared over n (Days 4, 15).
Covariance
Joint movement; a regression slope is Cov(X, Y) / Var(X) (Days 11, 22).
Conditional probability
P(A | B) with Bayes; the prior is the base rate (Day 13).
Distributions
A model of the process: binomial, Poisson, normal, heavy-tailed.
Both counts average 4 no-shows. Ten or more: 0.8% of flights under Poisson, 4.6% when the rate varies by flight (exact).
The diagnostic: plot each route's variance against its mean. Poisson routes scatter around variance = mean; mixed-rate routes follow mu + mu^2/k and drift further above it as the mean grows.
Where it breaks: a buffer set at the Poisson 95th percentile (8 no-shows) covers 97.9% of Poisson flights but only 92.7% of overdispersed ones; their own 95th percentile is 9.
Left and right: exact probabilities (Poisson mean 4; negative binomial mean 4, k = 4, variance 8). Middle: 40 simulated routes with means 2-8, 150 flights each, per model (seeded).
- Five building blocks, each reused later: random variable and expected value, Variance and standard error, Covariance, conditional probability with the base rate, and distributions.
- When counts vary more than a Poisson model allows, they are overdispersed; a statistic's spread over repeated samples is its sampling distribution (Day 4).
- Details are in the pictures above and in the optional technical depth below.
Assumptions
Sensitivity and false-alarm rate measured in a pilot hold in the deployment population; if fraudsters adapt or the case mix differs, the posterior changes.
The prevalence used as the prior is the one for the population being scored, not for a review sample enriched with known cases.
Poisson and binomial assume independent events with a common rate; dependence or varying rates widen the tails.
Application and workplace decision
Run the code. It:
- computes the fraud screen's posterior with natural frequencies;
- checks it by simulation;
- compares Poisson no-show counts with counts whose rate varies by flight;
- compares the spread of a 25-flight average with one flight.
Before running, guess how much more often an overdispersed flight sees 10 or more no-shows than the Poisson model predicts.
Python
import numpy as np
from scipy import stats
# 1) Bayes with natural frequencies: a fraud screen on 100,000 card payments (synthetic rates)
n, prevalence, sensitivity, specificity = 100_000, 0.002, 0.95, 0.98
fraud = n * prevalence
true_alerts = fraud * sensitivity
false_alerts = (n - fraud) * (1 - specificity)
print(f"fraud {fraud:.0f} | true alerts {true_alerts:.0f} | false alerts {false_alerts:.0f}")
print(f"P(fraud | alert) = {true_alerts / (true_alerts + false_alerts):.3f}")
# 2) The same screen simulated, as a check on the arithmetic
rng = np.random.default_rng(3)
is_fraud = rng.random(n) < prevalence
alert = np.where(is_fraud, rng.random(n) < sensitivity, rng.random(n) > specificity)
print(f"simulated P(fraud | alert) = {is_fraud[alert].mean():.3f} from {alert.sum()} alerts")
# 3) No-shows per flight: Poisson implies variance = mean; real counts are often overdispersed
lam = 4.0
poisson = rng.poisson(lam, 20_000)
mixed = rng.poisson(lam * rng.gamma(4.0, 1 / 4.0, 20_000)) # the rate itself varies by flight
for name, x in [("poisson", poisson), ("overdispersed", mixed)]:
print(f"{name:13s} mean {x.mean():.2f} var {x.var():.2f} P(no-shows >= 10) {np.mean(x >= 10):.4f}")
print(f"Poisson formula P(>= 10) = {stats.poisson.sf(9, lam):.4f}")
# 4) Sampling distribution: a 25-flight average varies far less than one flight
means = rng.poisson(lam, (5_000, 25)).mean(axis=1)
print(f"sd one flight {np.sqrt(lam):.2f} | sd of 25-flight mean {means.std():.3f} (theory {np.sqrt(lam / 25):.3f})")
Mini-assessment
Technical depth
Definitions first: moments and the spread of an averageExpectation, variance, covariance and the standard error defined as math, checked against the Poisson run, and why averages calm down as root n.
- the probability-weighted average of outcomes: the long-run mean, and what squared-error models estimate
- expected squared distance from the mean; its square root (sd) is back in the units of Y
- the joint version, E[XY] - E[X]E[Y]; a regression slope is Cov(X,Y)/Var(X) (Day 11)
- the sd of a mean of n INDEPENDENT values; correlated values shrink slower than root n
Check on a count with mean 4 (Poisson): E = 4, Var = 4, and a 25-flight mean has se 2/5 = 0.40, exactly what today's simulation measures.
Random variable
A numeric outcome; its expected value is what squared-error models estimate (Days 11, 16).
Variance
Spread around the mean; an average of n values has variance sigma squared over n (Days 4, 15).
Covariance
Joint movement; a regression slope is Cov(X, Y) / Var(X) (Days 11, 22).
Conditional probability
P(A | B) with Bayes; the prior is the base rate (Day 13).
Distributions
A model of the process: binomial, Poisson, normal, heavy-tailed.
- scaling rule: the 1/n in front comes out squared
- = Var(X) + Var(Y) for INDEPENDENT X and Y: spreads add in variance, never in sd
- one Poisson flight has Var 4; the 25-flight mean has Var 4/25 = 0.16, sd 0.40, the simulated value
- if every pair is correlated rho: with rho = 0.1 the same mean has sd 0.74, not 0.40
Independence is the load-bearing assumption. Handle times within one call-centre shift, or claims from one storm, are correlated, so their average narrows slower than root n.
Exact values for Poisson no-shows with mean 4. Quadrupling n halves the spread; Day 4 builds confidence intervals on this curve.
Bayes derived, then worked by handThe theorem derived from the definition of conditional probability, the fraud numbers plugged in, and a second posterior worked by hand on a hospital ward.
- the definition of conditional probability: the share of A-cases that are also F-cases
- the joint, written the other way round: P(A | F) P(F) - the same definition applied to A given F
- law of total probability: alerts come from fraud cases or from legitimate ones, nowhere else
There is no extra axiom in Bayes' theorem: it is the definition of conditional probability read in both directions, with the denominator expanded. The tree computes the same three quantities as counts.
- prior: the base rate of fraud in the population you score
- sensitivity: share of fraud the screen flags
- false-alarm rate: 1 minus specificity
- posterior: share of alerts that are fraud (precision)
Write the three inputs
Prevalence 2% of ward patients deteriorate; sensitivity 85%; false-alarm rate 15% of the rest.
Count per 1,000 patients
20 true cases, 17 of them alert. 980 others, 147 of them alert.
Form the posterior
P(case | alert) = 17 / (17 + 147) = 17/164 = 10.4%.
Check with the formula
0.85 x 0.02 / (0.85 x 0.02 + 0.15 x 0.98) = 0.017 / 0.164 = 10.4%. Same number.
Stress the prior
On a high-risk unit with prevalence 5%, the same screen gives 0.0425 / 0.185 = 23%: the screen did not change.
Verified in Python. One alert in ten is real - the arithmetic behind alarm fatigue, and why the threshold must come from review capacity and costs.
Sensitivity 95% and specificity 98% in every row; only the base rate changes. Quote a precision only with the population it was measured on.
Bayes as odds, and chaining evidenceadvancedThe odds form, the likelihood ratio an alert needs, and when multiplying two signals double-counts.
- prior odds: 2 fraud per 998 legitimate payments
- likelihood ratio of an alert: 0.95 / 0.02 = 47.5, a property of the screen
- posterior odds; as a probability 0.095 / 1.095 = 8.7%, the same answer as the tree
A second, conditionally independent signal multiplies the odds again by its own likelihood ratio. If the two signals share causes, multiplying double-counts the evidence: two rules that fire on the same device fingerprint are one piece of evidence, not two.
Posterior odds pass 1 only when the likelihood ratio exceeds the inverse of the prior odds: rare events demand extraordinary evidence. The likelihood ratio travels between populations (if the screen's rates do); the posterior never does.
- Prior odds2 : 998 at a 0.2% fraud rate
- x LR of the alert0.95 / 0.02 = 47.5
- Independent given class?a second signal
- no: to Model them jointly
yes - x LR of signal 2posterior odds multiply again
- Model them jointlyshared cause: same device fingerprint
After the first alert the odds are 2/998 x 47.5 = 0.095, a probability of 8.7%. A second signal that shares causes with the first must be scored together with it; multiplying both ratios counts one piece of evidence twice.
Which error rate to improveWhy specificity moves precision far more than sensitivity when the event is rare.
| Screen | Alerts per 100,000 | Share of alerts that are fraud |
|---|---|---|
| Sensitivity 95%, specificity 98% | 2,186 | 8.7% |
| Sensitivity 99%, specificity 98% | 2,194 | 9.0% |
| Sensitivity 95%, specificity 99% | 1,188 | 16.0% |
| Sensitivity 95%, specificity 99.8% | 390 | 48.8% |
With rare events, false alarms come from the huge legitimate group, so specificity moves precision far more than sensitivity.
Choosing a distribution, and checking its tailThe process each family assumes, its variance, the variance-to-mean diagnostic and the overdispersion formula.
| Criterion | Binomial | Poisson | Normal | Heavy-tailed |
|---|---|---|---|---|
| Process | Successes in n independent trials, same probability | Counts of many independent rare events | Sums and averages of many small effects | A few values dominate the total |
| Variance | n p (1 - p) | Equal to the mean | A free parameter | Large; the sample mean is unstable |
| Example | Claims this year among 500 insured homes | Defects per shift; no-shows per flight in today's code | Average handle time over 100 support tickets | Revenue per customer, insurance claims |
| Breaks when | Trials are dependent (one storm, many claims) or probabilities differ | Events cluster (a bad batch, a group booking); rates differ by unit | Few values, heavy tails, hard bounds | Treated as normal: risk understated |
- mean count per flight (4 in today's run)
- how much the rate varies by flight; k = 4 gives variance 4 + 16/4 = 8
- the rate is the same on every flight and the Poisson model returns
A Poisson count whose rate follows a gamma distribution is negative binomial. Today's simulation draws exactly this and measures variance 8.1.
- Counts per segmentroutes, shifts, cells, stores
- variance / mean ~ 1Poisson is plausibleone rate per segment
- ratio 2 or moreOverdispersedrates vary; tail much fatter
- fit k per segmentNegative binomialVar = mu + mu^2 / k
Dispersion often grows with the mean (variance = mu + mu^2/k), so the busiest segments (large stores, busy cells, busy routes) are where Poisson quantiles fail first. The same check applies to defects per shift (manufacturing) and faults per cell (telecom).
From flags to a review queueTurning a vendor screen into a capacity-limited decision with diminishing returns per slot.
Ask for scores
A yes/no flag fixes one threshold; a score lets you choose it.
Measure on your traffic
False-alarm rate on payments like yours, not on the pilot sample.
Rank and cut
Review the 500 highest scores each day: capacity sets the threshold.
Price the misses
Compare the cost of a missed fraud with the cost of a review (Day 13).
Monitor the base rate
If fraud prevalence moves, precision moves with it.
The vendor cut flags about 2,186 payments a day, catching 190 of 200 fraud cases. Raising the cut to fill 500 slots still catches about 168: the extra alerts were almost all legitimate.
Capacity sets the threshold. Each extra review slot catches less fraud than the one before, while precision keeps falling: price the last slot against the cost of a missed fraud (Day 13).
Computed, not simulated: a binormal score model calibrated to the vendor figures (95% sensitivity and 98% specificity), 100,000 payments a day at 0.2% fraud. Real score distributions are rarely normal; the shape of the trade-off, not the exact counts, is the point.
Library conventions that change the numbersadvancedddof defaults, survival functions, the negative binomial parametrization and integer quantiles.
| Call | Convention | Trap |
|---|---|---|
| np.var(x), np.std(x) | ddof=0: divides by n | smaller than pandas on the same data |
| Series.var(), Series.std() | ddof=1: divides by n - 1 | mixing the two in one report |
| stats.poisson.sf(9, mu) | P(Y > 9), that is P(Y >= 10) | off-by-one when you want 'at least 10' |
| stats.nbinom(n, p) | n = k, p = k / (k + mu) gives mean mu, variance mu + mu^2 / k | passing the mean as p |
| stats.poisson.ppf(0.95, 4) | smallest y with P(Y <= y) >= 0.95: 8 | a quantile of a count is an integer, coverage is not exactly 95% |
Each row was checked against the installed versions. The nbinom row reproduces mean 4, variance 8 and P(Y >= 10) = 4.6%.
Bookkeeping trapsFour mistakes that produce confident wrong numbers.
Flipped conditional
Reading P(alert | fraud) as P(fraud | alert).
Wrong population
A prior from an enriched sample instead of live traffic.
Mean-only fit
A Poisson model that matches the mean but misses the tail.
Hidden dependence
Claims from one storm, or a group booking, arrive together; independence fails.
Memory anchor
Count people, not percentages: the base rate decides what an alert means, and the tail decides what a count model costs.
Useful for
Sizing alert queues (fraud, maintenance, no-show, anomaly alerts) before launch.
Explaining to stakeholders why a "95% accurate" screen yields mostly false alarms.
Choosing a count model and checking its variance and tail against data before using its quantiles.
Not suitable for
Estimating effects of interventions - a conditional probability describes association in the observed population (Day 5).
Situations where the rates themselves are unknown and unmeasured in your population; the arithmetic is exact, the inputs are not.
Traps
Quoting pilot precision for production
Pilot samples are often enriched with known positives. Precision and predicted alert volumes then look far better than live traffic delivers.
How to avoid it
Recompute precision with the live base rate, or report sensitivity and false-alarm rate separately and state the population.
Fitting the mean and trusting the tail
A Poisson model matched to the average count (no-shows per flight, claims per storm week, faults per telecom cell) understates how often extreme counts occur when rates vary between units or events cluster.
How to avoid it
Compare the sample variance with the mean and check tail frequencies before using the model's quantiles for decisions.
Senior reasoning
A senior writes the population, the base rate and the conditioning event before any number. They translate rates into counts per day so operations can see the queue, separate properties of the instrument (sensitivity, false-alarm rate) from properties of the population (prevalence), and ask whether the instrument's rates were measured where it will be used. For distributions, they check the variance and tail against data, because decisions such as overbooking and reserves depend on extremes, not on the mean.
Interview question
"A test is 99% accurate and the condition affects 1 in 1,000 people. You test positive. What is the chance you have it?" Walk through it.
Recall cards scheduled after completion
- Why does an accurate screen for a rare event produce mostly false alarms? Give the natural-frequency calculation.
- What does overdispersion mean for count data, and what is the standard error of a mean of n independent values?
Prerequisites and related concepts
- Cost of being wrong · day 1
- Target definition · day 2
- Random variables and expectation
- Variance and covariance
- Conditional probability and Bayes
- Common distributions
- Sampling distributions
- Confidence intervals · day 4
- Logistic regression and log loss · day 11
- Class imbalance · day 13
- Thresholds from cost matrices · day 13
- Forecast decision economics · day 26
- Autocorrelation · day 22
References
- How to Improve Bayesian Reasoning Without Instruction: Frequency Formatscore
Bayesian problems are solved far more often when presented as natural frequencies than as single-event probabilities - the basis for teaching Bayes with counts. (Abstract and the frequency-format studies)
- NIST/SEMATECH e-Handbook of Statistical Methodsoptional
Reference descriptions of the normal, binomial, Poisson, lognormal and other distributions with their shapes and typical uses. (1.3.6 Probability Distributions (1.3.6.6 Gallery of Distributions))
- Practical Statistics for Data Scientists: 50+ Essential Concepts Using R and Python (2nd edition)optional
Applied treatment of sampling distributions, the standard error and common distributions for analysts. (Ch. 2 Data and Sampling Distributions)
Optional sections
Deep dive
Bayes as odds. Posterior odds = prior odds x likelihood ratio. Fraud prior odds are 2:998; the likelihood ratio of an alert is 0.95 / 0.02 = 47.5; posterior odds are about 0.095, a probability of about 8.7%. The odds form makes chaining independent evidence easy - multiply by each test's likelihood ratio - and it shows why a rare event needs a very high likelihood ratio before an alert becomes more likely right than wrong.
Why the mean, why the median. The constant that minimizes expected squared error is the mean; the constant that minimizes expected absolute error is the median. That is why MSE-trained models estimate conditional means and MAE-trained models conditional medians (Day 16), and why heavy tails pull the two apart.
Heavy tails. In lognormal or Pareto-like data (revenue per customer, insurance claims, outage durations) a few observations carry much of the total. The sample mean converges slowly and jumps when a large value arrives; medians, trimmed means or modeling the log are often more stable, but the business may still need the mean (total revenue). Decide by the decision, then quantify the uncertainty (Day 4 bootstrap).
Overdispersed counts. A Poisson rate that itself varies (here a gamma-distributed multiplier) gives a negative binomial distribution, with variance . With and the variance is 8, matching the simulation. Day 26 uses such tails to set overbooking quantiles.
Covariance traps. Covariance measures linear co-movement only: a U-shaped relationship can have zero covariance. Correlation rescales covariance to [-1, 1] but still says nothing about cause (Day 5).
Interview practice
An emergency department's model predicts a mean of 4 ambulance arrivals per hour on weekday evenings, and the staffing lead wants enough trolleys and nurses to cover the 95th percentile of arrivals. Which distribution would you use, and how would you check it?