Core day 20 of 60 · Week 4 · Part I · evaluation depth

ML Checkpoint

A churn model scores AUC 0.747 against the incumbent rule's 0.605 in cross-validation, and the team asks to switch on retention offers for the whole base. You have one validation run, a segment split and an offer cost. Which evidence decides whether to ship, and what exactly do you recommend?

By the end: You review a model that beats its baseline offline against six checks (decision and baseline, validation design, operating-point value, slices, causal claim, production readiness) and defend a scoped ship decision with numbers.

At a glance

A model earns a budget through a chain of six checks with a number behind each; the weakest link, not the headline metric, sets the release.

Baseline and decision
Name the action, the incumbent and the price before reading any metric.
Split like deployment
Random folds gave +0.14 AUC; the time-ordered holdout gave a tie.
Price the policy
Top-5% precision against break-even precision c / (s v), then net value in euros.
Slices and calibration
Postpaid pays, prepaid is mis-scored; one aggregate hides both.
Ranking is not effect
The save rate is causal: measure it with a randomized control in a scoped canary.

Recall · 5 min

Checkpoint warm-up, closed book, two minutes each: (1) what three things must a validation split copy from deployment (Day 15)? (2) give two conditions under which MAPE misleads (Day 16). (3) name the three monitor signals and say which one needs labels (Day 19). (4) why is permutation importance not an intervention effect (Day 17)? Then clear your due review cards.

Open due reviews

Concept · 10 min

By now you have a dozen tools: baselines, splits, metrics, slices, explanations and monitors. A checkpoint asks one question with all of them: would you stake a budget on this model? The answer is never only "the AUC is higher". It is a short chain of checks, each with a number and a comparison, and the weakest link sets the release.

One aggregate hides a slice that is systematically wrong, and ranking by error rate points at the wrong one

Predict first

The model beats the rule by 0.14 AUC in random CV and ties it on a time-ordered holdout. Which number is the better estimate of production?

Text description of the figure

The Day 18 slicer on a seeded airline forecast table, used here as a practice run of the same habit on a different problem. A switch picks the slicing dimension (route, market, weekday, horizon) and the metric (WAPE, MAE, bias); a bar chart compares each slice with the overall line and a table ranks slices by volume times error and by error rate. The churn case itself is sliced in the script output, by segment, not in this widget.

Six checks, in order Sequence

1 Decision and baseline
What changes, against which incumbent, at what price? Top 5% get an offer; rule baseline; EUR 15.
2 Validation design
Does the split copy deployment? Random 5-fold against a time-ordered holdout.
3 Operating point
Is the capacity-limited policy worth its cost? Net EUR against break-even precision.
4 Slices and calibration
Where does it fail? Postpaid versus prepaid; mean score versus observed rate.
5 Causal claim
Does the offer change outcomes for those flagged? Today the save rate is assumed.
6 Readiness
Data contract, monitors with label delay, canary, rollback and an owner (Day 19).

Each check gates the next; a failure points to the fix Diagram

Split copies deployment?
- yes: to Beats break-even?
no
Re-run on later monthsa time-ordered holdout
Beats break-even?top-5% precision
- yes: to Holds in the slices?
no
Change the policycost, k or the offer
Holds in the slices?with calibration
- yes: to Effect measured?
no
Scope to passing slicesrecalibrate
Effect measured?the save rate
- yes: to Scoped release
no
Measure it in a pilotrandomized, sized in advance
Scoped releasecontract, canary, rule as rollback

Check 1 has named the decision and baseline. Today the headline fails check 2, the overall policy fails check 3, only postpaid passes check 4, and the effect is unmeasured. The result is a scoped pilot that measures it; readiness (check 6) is part of the release itself.

The same model, two splits (AUC) Chart

Model, random 5-foldthe slide0.747 AUC
Rule, random 5-fold0.605 AUC
Model, time holdouttrain months 0-5, test 6-70.625 AUC
Rule, time holdout0.626 AUC

The 0.14 gap comes mostly from letting later months into training. On the later months the model and the rule tie.

Net EUR of offers to the top 5% on the time holdout Chart

All: model-2,955 EUR
All: rule-2,430 EUR
Postpaid: modelabout 1.5 standard errors above zero1,170 EUR
Postpaid: rule-1,080 EUR
Prepaid: modelmean score 0.158 against a rate of 0.053-360 EUR
Prepaid: rule-810 EUR

Assumed save rate 25%, margin EUR 300, offer EUR 15. Computed by the application script.

Break-even precision falls as the save rate rises Chart

Break-even c / (s v)
Postpaid top 5% (22.3%)
All customers top 5% (15.9%)

A policy pays where its precision line is above the curve. Postpaid clears it from a save rate of about 0.22; all customers only from about 0.31.

Key terms: Baseline, Cross-validation, Break-even precision, error slices, Causal effect and Canary release. The six checks and today's numbers are in the pictures above; the optional technical depth below adds decision maps for Days 11-19 (which model, validation, metric and diagnostic when).

Assumptions

The assumed save rate (25%), margin (EUR 300) and offer cost (EUR 15) are placeholders to replace with finance and a pilot; the decision rule is what transfers.
The time-ordered holdout is the closest available copy of deployment, with months 6 and 7 including a competitor promotion; a different later period could rank things differently.
Slice results on a few hundred flagged rows carry sampling noise, so each is read with an interval, not a point.

Application and workplace decision · 10 min

The script simulates 8 months of a telecom base (12,000 customers, synthetic), with a prepaid product launched in month 5 and a competitor promotion in months 6-7.
It scores the incumbent rule and a gradient-boosted model twice: random 5-fold CV, then train on months 0-5 and test on months 6-7.
It prices offers to the top 5%, overall and by segment.
Fill in the six-check table before you read the interpretation.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import KFold, cross_val_predict
rng = np.random.default_rng(20)
C, M = 12000, 8                                                    # customers x months (synthetic telecom base)
cust, month = np.repeat(np.arange(C), M), np.tile(np.arange(M), C)
u = rng.normal(0, 1, C)[cust]                                      # persistent customer propensity
prepaid = (rng.random(C) < 0.3)[cust]                              # prepaid product launches in month 5
calls = rng.poisson(0.4 + 0.3 * np.clip(u, 0, None))
end_soon = (~prepaid) & (rng.random(C * M) < 0.15)                 # contract ends within 30 days (postpaid only)
dip = rng.normal(0, 1, C * M) + 0.6 * u
season = np.array([0, 0.1, 0.3, 0.0, 0.4, 0.5, 1.0, 1.2])[month]   # competitor promotion in months 6-7
launch = np.where(prepaid, 0.3 * dip + 1.8 * (month == 5), 0.1 * dip)   # prepaid churn spikes once, at launch
y = rng.random(C * M) < 1 / (1 + np.exp(-(-4.4 + 0.5 * calls + 1.1 * end_soon + 0.6 * u + season + launch)))
df = pd.DataFrame({"cust": cust, "month": month, "tenure": rng.integers(1, 60, C)[cust] + month, "calls": calls,
                   "end_soon": end_soon.astype(int), "dip": dip, "prepaid": prepaid.astype(int), "y": y})
df = df[~((df.prepaid == 1) & (df.month < 5))]                     # no prepaid rows before launch
df = df[df.month <= df.cust.map(df[df.y].groupby("cust").month.min()).fillna(99)]   # churned customers leave
F, tr, te = ["month", "tenure", "calls", "end_soon", "dip", "prepaid"], df[df.month < 6], df[df.month >= 6]
rule = lambda d: (d.end_soon * 2 + (d.calls >= 2)).to_numpy() + rng.random(len(d)) * 0.01   # incumbent rule
gb = HistGradientBoostingClassifier(max_iter=100, random_state=0)
cv = cross_val_predict(gb, df[F], df.y, cv=KFold(5, shuffle=True, random_state=0), method="predict_proba")[:, 1]
p, r, yt, pp = gb.fit(tr[F], tr.y).predict_proba(te[F])[:, 1], rule(te), te.y.to_numpy(), te.prepaid.to_numpy() == 1
print(f"random 5-fold AUC: model {roc_auc_score(df.y, cv):.3f} vs rule {roc_auc_score(df.y, rule(df)):.3f}")
print(f"time holdout AUC:  model {roc_auc_score(yt, p):.3f} vs rule {roc_auc_score(yt, r):.3f}")
SAVE, VALUE, COST = 0.25, 300.0, 15.0                              # assumed: save rate, EUR margin kept, EUR per offer
def net(mask, s):                                                  # offers to the top 5% of the scored rows
    idx = np.argsort(-s[mask])[: int(0.05 * mask.sum())]
    return SAVE * VALUE * yt[mask][idx].sum() - COST * len(idx), yt[mask][idx].mean()
for name, mask in (("all", np.ones(len(yt), bool)), ("postpaid", ~pp), ("prepaid", pp)):
    v, prec = net(mask, p)
    print(f"{name:<9} n={mask.sum():5d} rate {yt[mask].mean():.3f} mean score {p[mask].mean():.3f} "
          f"AUC {roc_auc_score(yt[mask], p[mask]):.3f} | top-5% precision {prec:.2f} net EUR {v:,.0f} | rule net EUR {net(mask, r)[0]:,.0f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 40 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

Net value and break-even precision, derived7 minThe net value of acting on the top k, the group and marginal break-even rules derived from it, and six collection calls priced by hand.

Net value of acting on the top k Formula

\mathrm{Net}(k) = \sum_{i=1}^{k} \big(s\,v\,y_i - c\big) = s\,v\,\mathrm{TP}_k - c\,k, \qquad \mathbb{E}\big[\Delta\mathrm{Net}_{k}\big] = s\,v\,\pi_k - c

$y_i$: 1 if the i-th ranked customer would have had the bad outcome (churn, default, readmission) without the action
$\mathrm{TP}_k$: true positives among the k acted on: precision at k times k
$s\,v$: expected value of acting on one true positive: success probability (causal) times value kept
$c$: cost of one action, paid for every one of the k, positive or not
$\pi_k$: calibrated probability that the k-th customer is positive: the marginal precision

Everything that follows is algebra on this line. It breaks when the action also changes negatives (an offer that annoys a loyal customer), when values differ by row (use s_i v_i), or when actions interact across rows (call-centre capacity).

Derivation: from net value to the two break-even rules Sequence

Average per action
Divide Net(k) by k: s v (TP_k / k) - c = s v precision_k - c.
Group break-even
The policy as a whole pays when precision_k > c / (s v). This is break-even precision p*.
Marginal step
Adding customer k+1 changes expected net by s v pi_{k+1} - c, where pi is that customer's calibrated probability.
Marginal break-even
Keep adding while pi_{k+1} > c / (s v): the same p*, applied to the next customer, not to the group average.
The optimum
Net(k) peaks where pi crosses p*. The average precision there is still above p*, because the top of the list lifts it.
What it needs
The marginal rule needs calibrated probabilities; the group rule needs only counts. Both need s, which only an experiment gives.

The same algebra prices fraud reviews, collection calls, retention offers and outreach to patients. The group rule survives uncalibrated scores; the marginal rule needs calibration (Day 13), per segment if segments differ.

Break-even precision Formula

p^{*} = \frac{c}{s\,v}

$c$: cost of one offer (EUR 15)
$s$: probability that the offer saves a flagged customer who would have churned (0.25 assumed)
$v$: margin kept when a customer is saved (EUR 300)
$p^{*}$: share of churners among those offered at which the average offer returns its cost

Net value by hand: six collection calls, ranked by calibrated risk Table

Rank	Calibrated P(default)	Marginal net 20 p - 5 (EUR)	Cumulative net (EUR)	Average p of those called
1	0.60	+7.0	7.0	0.60
2	0.45	+4.0	11.0	0.525
3	0.30	+1.0	12.0 (peak)	0.45
4	0.22	-0.6	11.4	0.39
5	0.15	-2.0	9.4	0.34
6	0.08	-3.4	6.0	0.30

A bank's collections team (illustrative): a call costs EUR 5, recovers EUR 100 with probability 0.2 if the customer would default, so s v = 20 and p* = 5 / 20 = 0.25. Stop after rank 3, the last p above 0.25; at rank 6 the group average (0.30) still clears p* while the last three calls lose money. Verified in Python.

Standard error of a difference: pilots and paired comparisonsadvanced7 minWhy variances add for independent arms, a randomized pilot worked by hand, and how pairing on the same origins shrinks the standard error.

Standard error of a difference between two means Formula

\mathrm{SE}(\bar x_1 - \bar x_0) = \sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_0^2}{n_0}}, \qquad \mathrm{SE}(\bar d) = \frac{s_d}{\sqrt{k}} \ \text{ for } k \text{ paired differences}

$\bar x_1, \bar x_0$: means in two independent groups, such as treated and control arms of a pilot
$\sigma^2 / n$: variance of one group's mean; for a rate p it is p (1 - p) / n
$+$: variances of independent quantities add, so the difference is noisier than either mean
$\bar d, s_d$: mean and standard deviation of paired differences on the same units (rows, folds, origins)

Derivation: Var(A - B) = Var A + Var B - 2 Cov(A, B). Independent arms have Cov = 0; pairing makes Cov large and positive, which is why paired comparisons are sharper.

A randomized outreach pilot, worked by hand Table

Arm	Patients	Readmitted within 30 days	Rate	Variance of the rate p(1 - p) / n
Outreach calls	400	60	0.15	0.15 x 0.85 / 400 = 0.000319
Control (no call)	400	80	0.20	0.20 x 0.80 / 400 = 0.000400
Difference	-	-	0.05 fewer	SE = sqrt(0.000719) = 0.027

Hospital readmissions (illustrative). 95% interval 0.05 +/- 1.96 x 0.027 = -0.003 to 0.103: it just includes zero. The SE falls like 1 / sqrt(n), so halving it needs four times the patients, about 1,600 per arm: plan the size before the pilot starts. With few events, use a Wilson interval or an exact test.

Paired versus unpaired: five rolling origins of a load forecast Table

Origin	Incumbent MAE (MW)	Candidate MAE (MW)	Difference
1	43	41	2
2	44	44	0
3	41	38	3
4	48	47	1
5	42	40	2
Mean and SE	43.6	42.0	1.6, paired SE 1.14 / sqrt 5 = 0.51

Energy load forecasting (illustrative). Paired: t = 3.1, p = 0.03 (4 degrees of freedom); as two independent samples, SE 1.99 and p = 0.44, because origins differ in difficulty (correlation 0.97). k-fold scores share training rows, so s_d / sqrt(k) understates their noise; time-ordered origins with gaps are closer to independent.

Same five origins: pairing cuts the standard error about fourfold Chart

SE, treated as two samplesorigin difficulty stays in the noise1.99 MW
SE, paired by origindifficulty cancels in each difference0.51 MW
Mean improvement1.6 MW

Computed from the table above. Compare models on the same rows or origins whenever you can; the paired SE answers whether one is better there.

How sure is +EUR 1,170? Formula

\mathrm{SE}(\text{net}) = s\,v\,n\,\sqrt{\hat p\,(1-\hat p)/n}

$n$: offers made (672 postpaid customers)
$\hat p$: observed precision among them (0.223)
$s\,v$: EUR 75 of value per churner reached

The standard error is about EUR 810, so +EUR 1,170 is roughly 1.5 standard errors above zero. A lead worth a pilot, not proof.

The case, check by check4 minToday's evidence in one table, the break-even rule, and how much noise sits in the one positive number.

The case, check by check Table

Check	Evidence	Verdict
1 Decision and baseline	Offers to the top 5%; incumbent rule; EUR 15 per offer	Defined
2 Validation design	Random 5-fold 0.747 vs 0.605; time holdout 0.625 vs 0.626	Headline gap does not survive
3 Operating point	Top-5% precision 16% vs break-even 20%; net -EUR 2,955 (rule -EUR 2,430)	Fails overall
4 Slices	Postpaid 22%, +EUR 1,170 (rule -EUR 1,080); prepaid 18%, -EUR 360	Postpaid only, with noise
5 Causal claim	Save rate of 25% assumed, not measured	Open: measure it
6 Readiness	Scores mis-calibrated in both segments; churn label needs 30 days; rule as rollback	Not yet

Check 3 applies the group break-even rule c / (s v); check 4's one positive number is read with its standard error, and at about 1.5 standard errors it earns a pilot, not a rollout. The weakest link sets the release: check 2 removes the headline and check 5 leaves the euros resting on an assumption.

Postpaid's +EUR 1,170, as value reached minus offers paid Bridge

Value of 150 churners reached11,250 EUR
672 offers at EUR 15paid for every offer, churner or not−10,080 EUR
Net value, postpaid top 5%1,170 EUR

s v TP minus c k with s v = EUR 75 (assumed save rate 0.25 x EUR 300) and 150 churners among 672 offers (precision 22.3%). Only the first bar rests on the assumed save rate; the offer bill is certain.

Slices, calibration and the effect3 minWhy prepaid and postpaid are mis-scored in opposite directions, and why the save rate needs its own experiment.

Why the two segments are scored wrongly in opposite directions Overview

One launch month
Prepaid rows exist from month 5 only, and that month carried a one-off launch churn spike.
The spike was learned
Prepaid mean score 0.158 against an observed rate of 0.053: over-scored.
Postpaid the reverse
Mean score 0.035 against a rate of 0.067: under-scored, yet ranked well enough to pay.
What to do
Hold prepaid until post-launch months accumulate; recalibrate on recent matured months.

One monotone recalibration map changes probabilities, not the ranking, so it does not move the top-5% numbers. Per-segment maps can reorder a pooled list, and both matter once a probability cut or an expected-value rule is used.

A risk ranking sees one axis; the save rate lives on the other Map

Would stay anyway

The offer's cost buys nothing; some 'saved' customers sit here.

Saved by the offer

The only cell that earns the save rate.

Pushed out by the offer

An offer that annoys a loyal customer changes a negative.

Lost either way

The most likely to leave may be the least persuadable; the ranking puts them first.

A risk ranking sorts customers left to right only, so it cannot tell the top-right cell from the bottom-right one. Uplift modelling (named on Day 5) ranks by expected effect instead of risk and needs randomized data to train and check it.

Which model when (Days 11-14)3 minA decision map from situation to the first model worth trying, and what each one cannot do.

Which model when (Days 11-14) Table

Situation	Start with	Watch out for
Always	The incumbent rule or Day 2 baseline, in business units	Beating a weak strawman instead of the real incumbent
Few rows; coefficients or monotone effects needed	Regularized linear or logistic regression (Day 11)	Coefficients are conditional associations, not effects
Interactions, thresholds, mixed column types	Random forest, then boosting with early stopping (Day 12)	No extrapolation beyond the training range
Rare positives, a capacity-limited action	A probabilistic classifier plus a cost-based cut (Day 13)	Resampling distorts probabilities; recalibrate
No label; segments wanted	k-means or similar, checked for stability (Day 14)	Clusters are fitted outputs; fit them inside the fold

From transparent to flexible (qualitative) Scale

Incumbent rulethe bar to beat; cheap to run
Regularized linear or logisticreadable; trends continue
Random forestfew knobs; flat outside the range
Gradient boostingmost capacity; stop early on validation

Which validation when (Day 15)3 minChoose the split from how the model will be used, and keep every fitted step inside it.

Which validation when (Day 15) Diagram

Scores later periods?
- yes: to Time-ordered split
no
Entities repeat?customer or store
- yes: to Group k-fold by entity
no
Shuffled k-foldonly for exchangeable rows
- to Tuning or selecting?
Time-ordered splitor rolling origins and their spread
- to Tuning or selecting?
Group k-fold by entityeach entity in one fold
Tuning or selecting?
- yes: to Nested CV or a test set
no
Report held-out score
Nested CV or a test setuntouched, used once

When time and groups both apply, combine them. The split copies how and when the model will be used, not what is convenient.

Which metric when (Days 13 and 16)3 minThe decision picks the metric; six common decisions and the metric that matches each.

Which metric when (Days 13 and 16) Table

Decision	Metric	Why not the obvious one
Act on the top k with a fixed budget	Precision at k and net value at that k	AUC averages over cut-offs you will never use
Probabilities used as numbers	Log loss plus a calibration check	A good ranking can be badly calibrated
Rare positives, ranking quality	Average precision	ROC AUC can look high when negatives dominate
Forecast with unequal over and under costs	Pinball loss at the cost-ratio quantile	MAE targets the median and ignores the cost ratio
Volumes across many series	WAPE or MASE	MAPE explodes near zero and is asymmetric
Classes at about 6% prevalence	Anything but accuracy	Predicting no churn for everyone scores about 94%

The metric follows the decision. A model metric is a means; the policy's value at the operating point is the decision number.

Which diagnostic when (Days 17-18)3 minImportance, effect plots, SHAP, slices, residuals and monitors, each with what it cannot show.

Which diagnostic when (Days 17-18) Table

Question	Use	It does not show
Which inputs does the model rely on?	Permutation importance on held-out rows	What changing the input in the world would do
How does the prediction move with one input?	PDP or ALE, with ICE curves for spread	A causal response; PDP misleads with correlated inputs
Why this one prediction?	SHAP values	An action effect; it describes the model
Where is it wrong?	Error slices ranked by volume times error	Why; test the root cause before fixing
Is the error structural?	Residuals against level, time and inputs (Days 11 and 18)	The fix; it locates the pattern
Is it still working?	Monitors by timing; matured outcomes by slice (Day 19)	Harm that has not yet reached its labels

Readiness and the way back (Day 19)2 minWhat must exist before the postpaid pilot receives a single offer.

Readiness for the postpaid pilot (Day 19) Sequence

Contract
Features as of scoring time; tenure, calls and contract end loaded before the run.
Recalibrate
Fit calibration on the latest matured months, per segment if needed.
Canary with control
Postpaid top 5%, with a randomized no-offer share inside the flagged group.
Stop rule
Agreed in advance: churn and net value in the offered arm against the control.
Monitors
Data quality at once, score drift daily, matured churn after its 30-day window.
Rollback
The incumbent rule stays registered and tested; one alias change reverts.

What would change the decision, and how many to offeradvanced5 minThe same offers priced at other save rates, how large a pilot must be to settle it, and why k is chosen on the marginal rate.

Net EUR of the model's top 5% at other save rates Table

Save rate	Break-even precision	All customers	Postpaid	Prepaid
0.10	50%	-9,795	-5,580	-2,700
0.20	25%	-5,235	-1,080	-1,140
0.25 (assumed)	20%	-2,955	+1,170	-360
0.30	16.7%	-675	+3,420	+420
0.40	12.5%	+3,885	+7,920	+1,980

Computed from the script's time-holdout counts (957, 672 and 284 offers). Net value is linear in s, so this is the first table a reviewer asks for: the sign of the decision turns on a number nobody has measured yet.

The decision turns on the save rate, so measure it Illustration

The model's top 5% pays once the save rate passes 0.17; the rule's needs 0.32. Bands: 95% noise in the observed precision.

If the true save rate were 0.3, a pilot needs about 1,000 customers per arm before its interval clears the break-even 0.17.

Synthetic base of 12,000 customers (the same generator as the profit-curve picture); the rule is a noisier ranking of the same risk. Model top-5% precision 29%. Assumed margin EUR 300, offer EUR 15. Pilot intervals: delta-method normal approximation for s = 1 - (churn if offered) / (churn if withheld).

Left: net value is linear in the save rate, and a weaker ranking needs a higher one to pay. Right: a randomized pilot's interval for the save rate shrinks only with the square root of its size, so size the pilot from the interval you need around break-even before it starts; one too small to exclude break-even only delays the decision.

How many to offer? Stop on the marginal churn rate Illustration

Net value peaks at 1.5%, 7.7% and 15.0% of the base for save rates 0.15, 0.3, 0.45. Past the peak every extra offer loses money.

At save rate 0.3, break-even is 16.7%. The next customers fall below it at 7.7% of the base, while the average over everyone offered is still 28%: stop on the marginal rate.

Synthetic base of 12,000 customers, churn rate 6.2%; the score is a noisy view of each customer's risk. Assumed margin EUR 300 and offer cost EUR 15; save rates are scenarios. Dotted grey line: the base churn rate.

Net value of offering from the top of the ranking down, for three save-rate scenarios. The peak is where the next customers' churn rate meets break-even, while the average over everyone offered is still far above it. A fixed 5% budget is a constraint, not an optimum: compare its net with the peak and report what the last offers earn.

Memory anchor

Decision, validation, operating point, slices, effect, readiness - the weakest link sets the release.

Useful for

Reviewing any "beats the baseline offline" claim before budget is committed: retention offers, collection calls, fraud reviews or patient outreach.
Structuring a ship / no-ship memo as six checks with a number behind each.
Deciding which weak area from Weeks 3 and 4 to spend the Sunday studio on.

Not suitable for

Exhaustive assessment; the checkpoint samples skills and the Week 4 Sunday studio extends it with a staged incident.
Real conclusions about telecom churn; the case is simulated so that the mechanisms are known.

Traps

Accepting the headline comparison

A cross-validated AUC against a weak baseline is taken as proof, although the split leaks later months and the metric says nothing about the policy that spends money.

How to avoid it

Re-run on a time-ordered holdout and price the operating point against break-even precision.

Treating a ranking as an effect

Correctly flagged churners (or defaulters a collections team calls) are counted as saved, so the value of the action is overstated by the whole unmeasured gap between risk and persuadability.

How to avoid it

Separate the risk model from the offer-effect estimate and measure the latter with a randomized holdout.

Senior reasoning

A senior reads a model claim as a chain: decision, baseline, validation, operating point, slices, effect, readiness. They find the weakest link first, size it in the business unit, and propose the smallest release that would measure the link they cannot verify. They state what result would change their mind, and who owns the rollback.

Interview question

"The data science team says its churn model beats the current rule by 15 AUC points. Do you approve the rollout?" How would you structure the answer?

Outline your answer first.

Recall cards scheduled after completion

What is break-even precision for an intervention, and what does it assume?
List the six checks for a ship decision, in order.

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction — Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, D. Sculley, 2017core
A rubric of 28 tests and monitoring needs for production readiness, the source for check 6.
Common pitfalls and recommended practices (scikit-learn User Guide, section 12) — scikit-learn developersoptional
Fit steps only on the training part inside a pipeline, the rule behind check 2. (Data leakage)
The Foundations of Cost-Sensitive Learning — Charles Elkan, 2001optional
Decisions follow from applying the cost matrix to probability estimates, which supports break-even precision and the calibration caveat.
Probability calibration (scikit-learn User Guide 1.16) — scikit-learn developersoptional
Calibration curves and recalibration methods for the mean-score-versus-rate comparison in check 4.
Canarying Releases (The Site Reliability Workbook, chapter 16) — Alec Warner, Stepan Davidovic et al., Googleoptional
A canary is a partial, time-limited deployment judged against a control with stop criteria set in advance.

Optional sections

Deep dive · 10 min

Why random folds inflate this claim. Random folds put rows from the promotion months and from after the prepaid launch into training, so the model can use month and prepaid to learn what happens there; a time-ordered split cannot. The month column is only part of the story: the post-launch behaviour of prepaid customers is the other part. The same mechanism appears in any panel where the label rate moves with time. The check is cheap: train on the past, test on the next block, and report both numbers.

Uncertainty at the operating point. Net value is $s\,v\,TP - c\,n$ for $n$ offers. With precision $\hat p$ over $n$ offers, the standard error of the net value is $s\,v\,n\sqrt{\hat p(1-\hat p)/n}$ . At $n=672$ postpaid offers and $\hat p=0.22$ with $s\,v = 75$ this is about EUR 800. A net of +EUR 1,170 is therefore about 1.5 standard errors above zero: encouraging, not conclusive.

Risk is not uplift. The model ranks customers by their chance of leaving. An offer helps only where it changes the decision of someone who would otherwise leave. Customers most likely to leave may be the least persuadable, and some of the "saved" would have stayed anyway. Uplift modelling and randomized holdouts (Day 5, Day 29) address this; the ranking model alone does not.

What would change the recommendation. A pilot save rate of 40% would move break-even precision to 12.5%, and the top-5% policy (16% precision overall) would become profitable. A save rate below 10% moves it above 50%, and no slice clears it at this cost.

Interview practice · 10 min

You present "do not roll out; run a postpaid canary" and the VP says the competitor is moving and you are being too cautious. How do you respond?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-11 · depends on scikit-learn, numpy, pandas