Core day 19 of 60 · Week 4 · Part I · application depth

MLOps Foundations

A no-show model has run for eight weeks. Last night the input monitor fired (PSI 0.5 on lead time) and the on-call engineer wants to retrain. Which hops from data to decision should you be able to trace, which signal shows the model is hurting decisions, and why can a monitor be loud and wrong, or silent and wrong?

By the end: You trace a prediction from data to decision, say what must be recorded and who owns each hop, distinguish training-serving skew from data drift and concept drift, and choose monitors and a rollback path when labels arrive late.

At a glance

A production model is a chain of hops from data to decision; make each one recorded, released in stages, monitored by timing and reversible.

A path
Data, features, training, registry, serving, decision: each hop has an owner and a record.
Skew is a bug
Same facts, two feature definitions: visible from the first request and fixed in code.
Drift is not harm
Input alarms show movement; only matured outcomes show whether decisions got worse.
Release in stages
Offline gate, shadow, then a canary against a control with stop rules agreed in advance.
Roll back by alias
Production points at an alias; reverting re-points it to a retained, tested version.

Recall · 5 min

Without notes: (1) which timestamps does a data contract need besides the value itself (Day 2)? (2) why must a fitted preprocessing object be shipped with the model (Day 9)? (3) what is the first test for a segment whose error suddenly rises (Day 18)? Then clear your due review cards.

Open due reviews

Concept · 10 min

A model in production is a system, not a file: source tables, feature code, a trained artifact, a serving path and a decision. Each hop has an owner and can fail without an error message, because a broken model still outputs plausible numbers. MLOps is making every hop reproducible, observable and reversible.

Input drift fires early but may be harmless; concept drift is silent until labels arrive

Predict first

Only the relationship between lead time and no-show changes; the lead-time distribution stays the same. Which signal moves first?

Text description of the figure

A daily timeline (150 days, drift from day 50) with a drift-type switch (covariate, label, concept), controls for drift speed, label delay in days and PSI threshold. Four panels show input PSI, input KS, prediction PSI and delayed log loss, and a readout names which alert fires first. With covariate drift the input monitors fire first; with concept drift only the delayed performance panel moves, after the label delay.

What a production model needs besides the model Overview

A contract
Inputs with meaning, event and load time, freshness and an owner.
One feature definition
Shared by training and serving, computed as of decision time.
A registry and an alias
Versioned models; production points at an alias, so rollback is one change.
A staged release
Shadow, then a canary against a control, with stop rules set in advance.
Monitors with known timing
Fast monitors see inputs; only matured outcomes see harm.
A way back
The previous version retained and tested, and a fallback rule when all else fails.

The path from data to decision, and back Diagram

Source tablescontract: schema, freshness, load time
Feature pipelineone definition, as of decision time
- to Training pipeline
Harm on matured labels?
retrain
Training pipelineversioned data, code, config
- to Passes the gate?
Monitorsquality, drift, matured outcomes
- back to Harm on matured labels?
Passes the gate?beats incumbent
yes
Registryversions + alias champion
alias
Servinglogs served features, scores, version
- logs: back to Monitors
Decisionapprove, price, offer, flag

The gate compares with the incumbent overall and by slice; an alert without matured harm is fixed in code, annotated or rolled back, not retrained. Serving calls the same feature code as training. Rollback is not a retrain; it re-points the registry alias to the previous version.

Four things that look like "the model got worse" Comparison

Criterion	Training-serving skew	Data driftcovariate shift	Label shift	Concept drift
What changed	The code: one feature, two definitions	The input population	The outcome rate; inputs within each class unchanged	The input-to-outcome relationship
When it shows	From the first request	After launch, gradually or at a step	After launch	After launch; inputs may look unchanged
Which signal sees it	Served features compared with training features for the same rows	Input PSI or KS; prediction drift	Mean score against the observed rate	Only matured outcomes, or a validated proxy
What fixes it	One shared definition; log served features	Often nothing: check matured outcomes first	Prior or threshold adjustment (Day 13)	Retrain on matured labels, through shadow and canary

Skew is a defect in your pipeline; the other three are changes in the world. Only the last needs a new model.

Monitoring in layers, fastest first Stack

Service healthLatency, errors, throughput. Seconds. Says the system runs, not that it is right.
Data qualitySchema, freshness, missing rate and ranges against the contract. Minutes.
Input driftPSI or KS per feature against a reference window. Hours. Blind to concept drift.
Prediction driftScore distribution and flagged share. Hours. Can miss shifts that cancel out.
Proxy outcomeA short-delay signal, such as a first payment or a first login, validated against the true label. Days.
Matured outcomeCalibration and error by slice once labels arrive. The first layer that measures harm directly.
Business resultCost per decision against a control group. Slowest and closest to value.

While labels are late, validate a short-delay proxy against matured labels and keep immature weeks out of training. A small random control that never receives the model's decision keeps outcome comparisons clean. Read drift beside feature importance (Day 17) and slice performance (Day 18): a drifted feature the model ignores is noise.

Loud and harmless, silent and harmful Illustration

Covariate case: PSI crosses 0.25 in week 12. Concept case: never; its highest weekly PSI is 0.020.

Concept case: the model over-predicts no-shows from week 10, but week 10's labels mature only in week 13. Shaded: harm that no monitor can see yet.

Synthetic weeks of 1,000 bookings, generated as in the application script (lead time Gamma(4, 7.5); change from week 10; labels 3 weeks late). The "model" here is the week 0-5 rule itself; PSI uses the app's drift helpers with 10 reference-quantile bins.

The two scenarios of the application, week by week. Input PSI fires only for the covariate case; the concept case shows only when its labels mature, three weeks after the harm starts. An input alarm says the population moved, not that the model is wrong, and green input monitors do not say it is right.

Key terms: Data contract, Training-serving skew, Data drift (covariate shift), Label shift (prior shift), Shadow deployment and Canary release. The path, the signals and the drift types are in the pictures above; PSI and KS, thresholds, release patterns, delayed labels and retraining triggers are in the optional depth below.

Monitor in the order signals arrive: data quality in minutes, input and score drift in hours, matured outcomes after the label delay.
Drift is not harm, and no drift is not health. Retrain on matured outcomes, through shadow and canary, with an alias you can flip back.

Assumptions

Labels mature on a known schedule; otherwise "recent" performance mixes mature and immature cases.
The reference window is representative; one taken after a change hides it.
Each monitor has an owner and a runbook; an unowned alert is a log line.

Application and workplace decision · 10 min

The script simulates 20 weeks of bookings for a no-show model trained on weeks 0-5. Labels arrive three weeks late.
Covariate case: lead times lengthen from week 10; the lead-time rule is unchanged.
Concept case: the rule weakens from week 10; the inputs look the same.
It prints the first PSI alert (threshold 0.25), KS, the mean score, the calibration gap (observed minus predicted rate) and when harm becomes visible. Predict both cases before you run it.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
from scipy.stats import ks_2samp
from sklearn.ensemble import HistGradientBoostingClassifier

DELAY, START = 3, 10                                              # labels arrive 3 weeks late; the change starts in week 10

def make_week(rng, w, scenario, n=2000):
    lead = rng.gamma(4, 7.5, n)
    if scenario == "covariate" and w >= START:
        lead *= 1 + 0.12 * (w - START + 1)                        # longer-lead bookings; the lead -> no-show rule is unchanged
    slope = 0.02 if scenario == "concept" and w >= START else 0.06    # a new fee rule weakens the lead-time effect
    return lead[:, None], (rng.random(n) < 1 / (1 + np.exp(-(-3.4 + slope * lead)))).astype(int)

def psi(ref, x, k=10):                                            # bins from the reference quantiles
    edges = np.quantile(ref, np.linspace(0, 1, k + 1)[1:-1])
    e, a = (np.bincount(np.searchsorted(edges, v), minlength=k) / len(v) for v in (ref, x))
    return float(np.sum((a - e) * np.log((a + 1e-6) / (e + 1e-6))))

for scenario in ("covariate", "concept"):
    rng = np.random.default_rng(19)
    data = [make_week(rng, w, scenario) for w in range(20)]
    ref_X, ref_y = np.vstack([d[0] for d in data[:6]]), np.concatenate([d[1] for d in data[:6]])
    model = HistGradientBoostingClassifier(max_iter=60, random_state=0).fit(ref_X, ref_y)
    score = lambda w: model.predict_proba(data[w][0])[:, 1].mean()    # prediction drift: available at once
    gap = lambda w: data[w][1].mean() - score(w)                      # observed - predicted rate: needs labels
    first = lambda f: next((w for w in range(6, 20) if f(w)), None)
    w_psi = first(lambda w: psi(ref_X[:, 0], data[w][0][:, 0]) >= 0.25)
    w_bad = first(lambda w: abs(gap(w)) > 0.05)
    ks = ks_2samp(ref_X[:, 0], data[19][0][:, 0]).statistic
    print(f"{scenario:<9}| PSI alert week {w_psi} | KS week 19 {ks:.2f} | mean score {score(5):.3f} -> {score(19):.3f}"
          f" | calibration gap week 19 {gap(19):+.3f}")
    print(f"{'':<9}| harm starts week {w_bad}, seen with labels in week {None if w_bad is None else w_bad + DELAY}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 42 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

PSI: definition, derivation as a divergence, a computation by handadvanced8 minThe formula, why it equals KL both ways and is never negative, four bins computed on paper, and how much PSI noise alone produces.

Population stability index Formula

\mathrm{PSI} = \sum_{i=1}^{k} (a_i - e_i)\,\ln\frac{a_i}{e_i}

$e_i$: share of the reference window in bin i (bins from reference quantiles)
$a_i$: share of the current window in the same bin
$k$: number of bins

Bins from reference quantiles keep every e_i equal, so no bin starts nearly empty; an empty current bin needs a small constant or the log is infinite. Thresholds are choices: Evidently documents KS up to 1,000 rows, normed Wasserstein above, and 0.1 for PSI.

Derivation: PSI is a symmetric divergence between two histograms Sequence

Two histograms
Cut the feature into k bins from reference quantiles. e_i = reference share in bin i, a_i = current share. Both sum to 1.
KL one way
KL(a || e) = sum of a_i ln(a_i / e_i): the extra log-loss of describing current data with the reference shares.
KL the other way
KL(e || a) = sum of e_i ln(e_i / a_i) = - sum of e_i ln(a_i / e_i).
Add them
KL(a || e) + KL(e || a) = sum of (a_i - e_i) ln(a_i / e_i) = PSI. It is the symmetric (Jeffreys) divergence.
Every term is positive
If a_i > e_i both factors are positive; if a_i < e_i both are negative. A bin that shrinks adds as much as one that grows.
Small changes
ln(a / e) is close to (a - e) / e, so PSI is close to the sum of (a_i - e_i)^2 / e_i: a chi-square statistic divided by n.
What it cannot do
It has no direction, no notion of which bins matter to the model, and it is never exactly zero on a finite sample.
Properties
Zero only when the histograms match; symmetric in the two windows; unchanged by monotone transforms; blind inside a bin.

Standard information-theory identities. The chi-square link gives the no-drift noise level in the equation further down.

PSI by hand: monthly data usage, four quartile bins Table

Bin (reference quartile)	Reference share e	This month: count, share a	ln(a / e)	Term (a - e) ln(a / e)
Lowest 25%	0.25	30 of 200, 0.15	ln 0.6 = -0.511	(-0.10)(-0.511) = 0.051
Second	0.25	40, 0.20	ln 0.8 = -0.223	(-0.05)(-0.223) = 0.011
Third	0.25	50, 0.25	ln 1 = 0	0
Highest 25%	0.25	80, 0.40	ln 1.6 = 0.470	(0.15)(0.470) = 0.071
Total	1.00	200, 1.00	-	PSI = 0.133

Telecom customers' monthly data usage (illustrative). KL one way is 0.067, the other 0.066; the chi-square approximation gives 0.140. Noise alone, with 200 current and 1,000 reference rows, would give about 3 x (1/200 + 1/1000) = 0.018. Verified in Python.

PSI bin by bin: four equal bins drift to 10, 20, 30 and 40% Bridge

Identical histograms0 PSI
Bin 1: 25% to 10%a shrinking bin adds a large term+0.137 PSI
Bin 2: 25% to 20%+0.011 PSI
Bin 3: 25% to 30%+0.009 PSI
Bin 4: 25% to 40%+0.071 PSI
Total PSI0.228 PSI

Every term is positive whichever way the share moves, so the bridge only climbs; PSI says that inputs moved, not that it matters.

How much PSI to expect when nothing changed Formula

\mathrm{PSI} \approx \sum_{i=1}^{k} \frac{(a_i - e_i)^{2}}{e_i}, \qquad \mathbb{E}\left[\mathrm{PSI} \mid \text{no drift}\right] \approx (k - 1)\left(\frac{1}{n} + \frac{1}{m}\right)

$(a_i - e_i)^{2} / e_i$: for small differences the log ratio is close to a chi-square term
$k$: number of bins; more bins
$n, m$: rows in the current and the reference window

A large-sample approximation. With 10 bins and 100 current rows it is about 0.09, close to a common 0.1 threshold; calibrate thresholds on no-drift windows of the same size.

PSI is never zero, and small windows inflate it Illustration

median over 80 windows (bar: full range)
95th percentile
dashed: approximation (k - 1)(1/n + 1/m)

Synthetic data: reference and current windows are both standard-normal draws, so every PSI here is pure sampling noise. Computed with the app's PSI helpers; the dashed line is a large-sample approximation, not a guarantee.

No drift at all, yet with 100 current rows the median PSI is 0.086 and 36% of windows reach 0.1. With 5,000 rows the 95th percentile is 0.008. Set thresholds per sample size, or test against the no-drift spread.

Both windows come from the same distribution. The PSI that sampling noise alone produces falls roughly as 1 / n, so a threshold that is quiet on a large store fires on a small one. Compute PSI on pairs of no-drift windows of the production size: an alert level below that noise fires every day.

KS: the largest gap between two empirical CDFs6 minThe statistic and its test, five values per window computed by hand, why its threshold shrinks with the window size, and how it differs from PSI.

The two-sample Kolmogorov-Smirnov statistic Formula

D = \sup_x \big|\, \hat F_{\mathrm{ref}}(x) - \hat F_{\mathrm{cur}}(x) \,\big|, \qquad \text{reject at level } \alpha \text{ if } D > c(\alpha)\sqrt{\tfrac{n + m}{n\,m}}

$\hat F(x)$: an empirical CDF: the share of the sample at or below x
$\sup_x$: the largest vertical gap between the two step functions, wherever it occurs
$n, m$: rows in the reference and the current window
$c(\alpha)$: about 1.36 at alpha = 0.05 (large-sample value); exact tables or scipy for small samples

No bins, no direction, and a p-value under the null of identical distributions. Its threshold shrinks like 1 / sqrt(n), which is its main property in monitoring.

KS by hand: five basket values before and after Table

Basket value x (EUR)	Reference ECDF (10, 20, 30, 40, 50)	Current ECDF (30, 40, 50, 60, 70)	Gap
10	0.2	0.0	0.2
20	0.4	0.0	0.4
30	0.6	0.2	0.4
40	0.8	0.4	0.4
50	1.0	0.6	0.4
60	1.0	0.8	0.2
70	1.0	1.0	0.0

Online-shop order values (illustrative). D = 0.4, first reached at x = 20. The 5% threshold for 5 and 5 rows is about 1.36 x sqrt(10 / 25) = 0.86, and scipy's exact p-value is far above 0.05: a whole shift of two steps is invisible at this size. Verified with scipy.stats.ks_2samp.

The KS threshold shrinks with window size: everything is significant at scale Table

Rows in each window	D needed for p below 0.05	Reading
5	about 0.86	Only gross changes are detectable
100	0.19	Matches a 4,000-run simulation of no drift (0.19)
10,000	0.019	A shift of 0.05 standard deviations gave D = 0.020, p = 0.03
1,000,000	0.0019	Practically any change, harmless or not, is flagged

Large-sample formula with c = 1.358, checked by simulation. Monitor the size of D (or PSI) against a tolerance agreed in advance, not the p-value.

PSI measures distance, KS runs a test: different blind spots Comparison

Criterion	PSI	KS
Answers	How far apart two histograms are	Whether two samples share one distribution
Needs bins	Yes, from reference quantiles	No
Sensitive to	Any bin's share; blind inside a bin	The middle; weak in the tails, so a top-1% change can pass
Window size	Small windows inflate it with noise	Large windows make harmless shifts significant
Categories	Works on categories directly	Continuous only: use chi-square or PSI

Monitor the size of either statistic against a tolerance agreed in advance; the p-value mostly measures the window size.

Many monitors: the family-wise false-alarm rate4 minDerive the chance that at least one of k quiet monitors fires, the union bound behind Bonferroni, and a worked production-line example.

Why many quiet monitors still alarm every day Formula

P(\text{at least one alarm}) = 1 - (1 - \alpha)^{k} \;\le\; k\,\alpha, \qquad \mathbb{E}[\text{alarms}] = k\,\alpha

$\alpha$: each monitor's false-alarm rate on a day when nothing changed
$k$: number of monitors evaluated that day
$(1 - \alpha)^{k}$: chance that all k stay quiet, if the monitors are independent
$k \alpha$: the union bound: true for dependent monitors too, and the basis of the Bonferroni correction alpha / k

Derivation: each monitor is quiet with probability 1 - alpha; independence multiplies them; the complement is at least one alarm. Correlated features alarm together, so the true rate sits between alpha and the bound. The same algebra applies to slices (Day 18), A/B metrics and repeated daily looks.

25 sensor monitors on a production line, worked Table

Rule	Per-monitor alpha	P(at least one false alarm a day)	Over a 30-day month
Each at 1%	0.01	1 - 0.99^25 = 0.22 (0.25 alarms a day)	1 - 0.78^30: almost certain, about 7 to 8 alarms
Bonferroni, 1% / 25	0.0004	1 - 0.9996^25 = 0.0100	About 0.3 alarms in the month

Manufacturing IoT, illustrative, independent monitors on a stable line. Bonferroni buys quiet days by losing sensitivity; a count of drifted features or a drift score weighted by the model's reliance (Day 17) keeps both. Prefer a few monitors tied to the decision: an alert nobody acts on trains the team to ignore the next one.

Monitoring layers and label delay4 minSeven layers from service health to business result, what each can see, and what to do while labels are late.

Monitoring in layers, fastest first Stack

Service healthLatency, errors, throughput. Seconds. Says the system runs, not that it is right.
Data qualitySchema, freshness, missing rate and ranges against the contract. Minutes.
Input driftPSI or KS per feature against a reference window. Hours. Blind to concept drift.
Prediction driftScore distribution and flagged share. Hours. Can miss shifts that cancel out.
Proxy outcomeA short-delay signal, such as a first payment or a first login, validated against the true label. Days.
Matured outcomeCalibration and error by slice once labels arrive. The first layer that measures harm directly.
Business resultCost per decision against a control group. Slowest and closest to value.

The lifecycle, hop by hop3 minWhat each hop from source table to decision must record, how it fails without an error message, and the loop the decision closes.

Each hop, what it must record, and how it fails silently Table

Hop	Records	Fails silently when
Source and contract	Schema, meaning, event and load time, freshness, owner	A supplier redefines a field or a feed lands late
Feature pipeline	Feature definitions and versions; as-of time of each value	Training and serving compute the feature differently
Training pipeline	Data snapshot, code commit, config, environment, seeds, metrics	A run cannot be reproduced or compared
Registry	Model version, lineage, alias, reason for promotion	Nobody knows which version is live or why
Serving	Model version, served features, score and time per row or request	A job fails and yesterday's scores are reused
Decision	Action taken, threshold, manual overrides	The model changes the data it will later learn from

Sculley et al. (2015) argue that much of the maintenance cost of ML systems sits in data dependencies, glue code and configuration rather than model code. Breck et al. (2017) turn these hops into 28 tests in four groups (data, model, infrastructure, monitoring) as a readiness checklist.

The decision changes the data the next model learns from Loop

Model scoreswho looks risky or promisingdrives ↓
Decision actscredit granted, offer sent, booking flaggedshapes ↓
Outcomes observedmostly for those the decision let throughbecome ↓
Next training setalready filtered by the old modelretrains ↓
↺ back to 1 · Model scores

A hidden feedback loop (Sculley et al., 2015): if the model changes who gets credit or an offer, the next training set reflects the model. A random holdout that never receives the model's decision shows what the outcomes would have been.

Batch or online serving3 minPick the serving pattern from two questions - does the decision wait, and how fresh must the features be.

Two questions pick the serving pattern Map

Online scoring

A request-time call reading batch-built features from a low-latency store.

Online plus streaming features

Request-time decisions on seconds-old facts: the most to run and test.

Batch scoring

A scheduled list from warehouse tables: simplest to test, re-run and roll back.

Frequent batch runs

A schedule tight enough for the freshness needed, still re-runnable.

Choose batch unless the decision waits for the score or needs facts fresher than the schedule.

Batch or online serving Comparison

Criterion	Batch scoring	Online scoring
Use when	The decision runs on a schedule, like a nightly list of customers to call	The decision waits for the answer, like a card authorization or an insurance quote
Latency budget	Minutes to hours	Milliseconds per request
Features come from	Warehouse tables as of the run time	The request plus a low-latency feature store
Typical failure	The job fails or runs twice; stale scores are served	Timeouts; features that differ from the training definition
Safeguard	Idempotent writes keyed by run date; a freshness check	A timeout with a fallback rule; logged served features

Choose batch unless the decision waits for the score. Batch is simpler to test, re-run and roll back.

Training-serving skew and point-in-time features4 minWhere skew comes from, the as-of rule that prevents it, and three habits that catch it.

Where training-serving skew comes from Overview

Two code paths
SQL for training, application code for serving: one feature, two definitions.
Late-arriving rows
Training reads the completed table; serving reads only what has loaded by scoring time.
Missing values
A missing income is zero in one path and the branch median in the other.
Overwritten lookups
A joined table is corrected after training and history is overwritten, not versioned.
Refit preprocessing
An encoder refit at serving instead of shipping the fitted object (Day 9).

Three habits catch it (Google's Rules of ML 29, 32 and 37): log the feature values used at serving and train from those logs where you can; reuse one feature implementation in both pipelines; regularly recompute training features for served rows and compare.

A point-in-time feature Formula

x_i(t) = f\left(\{\, r : \mathrm{event}_r \le t,\ \mathrm{loaded}_r \le t \,\}\right)

$x_i(t)$: the feature for entity i as it could be known at decision time t
$\mathrm{event}_r$: when the fact in row r happened (a payment missed last month)
$\mathrm{loaded}_r$: when row r became available to the scoring job
$f$: the one feature definition that both training and serving call

Filtering on event time alone gives training a complete history that serving never sees. Filtering on load time too reproduces serving.

Shadow, canary, A/B: releasing with gates4 minStages of a release, what each stage can and cannot tell you, how big a canary must be, and how the way back works.

A release in stages Sequence

Offline gate
Beats the incumbent on a time-ordered holdout, overall and per slice, on a frozen evaluation set.
Shadow
Scores live inputs; outputs are logged, not used. Tests plumbing, latency and agreement.
Canary
A small, time-limited share of decisions against a control, with stop rules agreed in advance.
Ramp
Widen the share in steps while the guard metrics hold.
Full release
The alias points at the new version; the old one stays registered and tested.
Record
Why it was promoted, on which evidence, and what would trigger a revert.

Shadow, canary and A/B test answer different questions Comparison

Criterion	Shadow	Canary	A/B test
Affects decisions	No: outputs are only logged	Yes, for a small share	Yes, in randomized arms
Answers	Does it run, how fast, and how often does it disagree with the incumbent?	Does it break something or cause visible harm quickly?	The effect on the business outcome, with an interval
Cannot answer	Any effect on outcomes; the incumbent still decides	Small effects: the sample is too small to detect them	Anything before its planned duration ends
Typical size and length	All traffic or a full batch, days	A few percent, hours to days	Sized for power, often weeks

A canary is a safety check against a control (Google SRE workbook), not a substitute for a sized experiment (Day 5).

What a canary can and cannot see Illustration

A 1-point rise in the bad-outcome rate: a 1% canary reaches 80% power after 29 days, a 20% canary after 2.0.

Days until 80% power, 5% canary. A 5-point jump shows within a day; a half-point harm needs an experiment sized for it, not a canary.

Computed, not simulated: power of a one-sided two-proportion z-test at the 5% level (normal approximation), base bad-outcome rate 10% (no-shows, defaults or returns), 20,000 decisions a day split between canary and control. Volumes and rates are illustrative assumptions.

Power of a canary against a large control to detect a rise in a bad-outcome rate (no-shows, defaults, returns). Breakage shows within a day; a small harm needs weeks, which is an experiment, not a canary.

Size the canary from the smallest harm worth stopping for Formula

n_{\text{canary}} \approx \frac{(z_{\alpha} + z_{\beta})^{2}\, p\,(1 - p)}{\delta^{2}}

$z_{\alpha}$: about 1.64 for a one-sided test at 5%
$z_{\beta}$: about 0.84 for 80% power
$p$: the bad-outcome rate under the incumbent
$\delta$: the smallest rise in that rate worth stopping for, chosen before launch

Rule of thumb against a control much larger than the canary. With p = 0.10 and delta = 1 point: 6.18 x 0.09 / 0.0001, about 5,600 canary rows, roughly 6 days at 1,000 rows a day, as in the demo. Pick the stop metric and delta first; share and length follow from them, not from habit.

Rollback is re-pointing an alias; fallback is a rule Diagram

Alias championwhat production reads
points at
Version 7new, in canary
Guard metric breached?
- no: to Ramp up
yes
Re-point to version 6retained, tested, same contract
- v6 unsafe too: to Fallback rule
Ramp upnext stage of the release
Fallback rulebaseline if inputs leave range or service fails

A rollback is only fast if the old version still runs on today's features. Test it like a release.

Retraining: triage, triggers and schedules3 minRoute every alert through the same questions, and choose what may trigger a retrain.

Triage an alert before you retrain Diagram

Alert fires
Contract or feed broken?
- yes: to Fix the pipeline
no
Harm on matured labels?
- no: to Annotate and keep
yes
Train a candidateon matured labels only
- to Shadow
Fix the pipelinea code defect, not drift
Annotate and keepdrift without harm
Shadowalias ready to revert
gates pass
Promote and record

Retraining: on a schedule or on a trigger Comparison

Criterion	On a schedule	On matured harm	On input drift
Acts on	The calendar: weekly or monthly	Calibration or error beyond tolerance on matured labels	A PSI or KS alarm
Strength	Predictable; the pipeline stays exercised	Acts on actual harm	The earliest signal
Risk	Retrains when nothing is wrong and is late when something is	Waits for the label delay	Acts on movement, not harm; use it as a prompt to check

Whatever triggers it, a retrained model is a new candidate and passes the same gates, shadow and canary.

Reproducibility, model cards and decision recordsadvanced3 minWhat to version so a run can be recovered, and what to write down so someone else can operate the model.

What to version so a run can be recovered Table

What	Example	Why
Data	Snapshot id, or query plus as-of time	Retrain on exactly the same rows
Code	Commit of feature and training code	Know which definition produced a score
Configuration	Hyperparameters, thresholds, feature list	Changing a threshold is a release too
Environment	Library versions, container image	The same code can give different results across versions
Randomness	Seeds; note non-deterministic steps	Tell a real change from run-to-run noise
Artifact	The fitted pipeline with preprocessing, and its input signature	Serve exactly what was evaluated (Day 9)

Reproducible means recoverable within a stated metric tolerance, not bit-identical; many stacks differ across hardware and thread counts. Gate releases on a frozen evaluation set with a tolerance sized to run-to-run noise, and record that tolerance with the run.

The paperwork that makes a model operable Overview

Intended use
The decision, its users and out-of-scope uses (model cards, Mitchell et al. 2019).
Metrics and evaluation data
Which split, which metric, against which baseline, overall and by slice.
Factors
Groups where performance may differ: segment, region, channel.
Caveats
Known failure modes, label delay, input ranges outside which it must not be used.
Decision record
Why this version was promoted, the alternatives, and what would trigger a revert.
Runbook
An owner, threshold and action for each monitor, and the rollback steps.

Registries such as MLflow's give each version lineage and an alias that production reads; the pattern matters, not the tool.

Memory anchor

Early monitors see inputs, late monitors see harm; shadow, canary, then an alias you can flip back.

Useful for

Reviewing a credit, fraud, churn or demand model before launch with a path, owners, monitors and a rollback plan.
Writing the monitoring section of a design document for a model with delayed labels.
Triaging an alert as pipeline defect, data drift, concept drift or noise.

Not suitable for

Choosing a vendor or a tool; the lesson fixes concepts and leaves tool selection to your constraints and the documentation.
Replacing a runbook; thresholds, owners and actions must be set for each model and decision.

Traps

Treating drift as failure

An input alarm triggers a retrain, although outcomes are unchanged. The team spends effort, risks a new defect and learns to ignore alerts.

How to avoid it

Pair every input monitor with a matured-outcome check and act on harm, not on movement alone.

Calling green monitors healthy

Input and score monitors stay flat while the relationship changes, and harm runs for the label delay or longer.

How to avoid it

Add outcome monitors with a stated delay, a short-delay proxy and a canary against a held-out control.

Senior reasoning

A senior starts from the decision and the label delay, lists what each monitor can and cannot see, and sets actions for each alert in advance: pipeline defects are fixed in code, harmless drift is annotated, harmful drift triggers a candidate through shadow and canary with a tested rollback. They record why each model version was promoted and prefer a system whose failures are visible over one that is slightly more accurate.

Interview question

"How would you monitor a churn model whose labels arrive 90 days after the prediction?" How would you structure the answer?

Outline your answer first.

Recall cards scheduled after completion

How do training-serving skew, data drift and concept drift differ, and how does each show up?
What do shadow, canary and rollback each give you, and what makes rollback fast?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Hidden Technical Debt in Machine Learning Systems — D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, Dan Dennison, 2015core
Names ML-specific maintenance risks (boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration and changes in the external world) that justify tracking, contracts and monitoring.
The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction — Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, D. Sculley, 2017core
A rubric of 28 specific tests and monitoring needs for production readiness, used as the checklist behind the release and monitoring steps.
ML Model Registry (MLflow documentation) — MLflow projectoptional
Describes model versions with lineage, aliases that production workloads can point at, tags and descriptions, which underlie promotion records and alias-based rollback.
GX Core overview (Great Expectations documentation) — Great Expectationsoptional
An Expectation is a verifiable assertion about data, run through validation definitions and checkpoints in production, matching the validate-before-train-and-score step.
Customize Data Drift (Evidently documentation) — Evidently AIoptional
Documents default drift methods and thresholds (KS up to 1,000 rows, normed Wasserstein above, PSI available with threshold 0.1) and shows that detected drift depends on method, sample size and threshold.
Canarying Releases (The Site Reliability Workbook, chapter 16) — Alec Warner, Stepan Davidovic et al., Googleoptional
Defines a canary as a partial, time-limited deployment evaluated against a control, with metric selection and rollback decisions made in advance.
Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift — Stephan Rabanser, Stephan Günnemann, Zachary Lipton, 2019optional
Treats shift detection as two-sample testing and separates detecting a shift from judging whether it is harmful.

Optional sections

Deep dive · 10 min

What "reproducible" means. Fix the recoverable inputs: code commit, data snapshot id, configuration, library versions and seeds. Many training stacks are not bit-exact across hardware or thread counts, so define success as a tolerance on the evaluation metric, and gate releases on a frozen golden set with a tolerance sized to run-to-run noise. Record the tolerance with the run.

Skew in practice. Typical sources are two code paths for one feature (SQL for training, application code for serving), a joined table that changes between training and serving (a late-arriving correction), and different handling of missing values. Rules of ML (Rules 29, 32 and 37) recommend logging served features, reusing code between training and serving, and measuring skew across stages. Compare logged serving features with the training pipeline run on the same rows.

Delayed labels. Options in order of cost: monitor input and score drift as early indicators; build a proxy outcome with a short delay (a check-in event, a cancellation within 24 hours) and validate it against the matured label; hold out a small control population that never receives the model's decision, so outcome comparisons are not confounded by the model's own actions; exclude immature labels when training.

PSI is a tool, not a verdict. It depends on bin choices (quantiles of the reference), is sensitive to empty bins (hence the small constant in the code), and says nothing about which feature matters to the model. Weigh each feature's drift by its importance (Day 17) and check performance on slices (Day 18).

Hidden feedback loops. If the model's output changes which data you collect (a lender observes repayments only for approved applicants), future training data reflect the model's own behavior. Sculley et al. list this as a hidden feedback loop; keep a randomized holdout to measure it.

Interview practice · 10 min

Production shows precision falling for a fraud model, and a colleague says "it is data drift, let us retrain". How do you respond?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-11 · depends on scikit-learn, scipy, mlflow, dvc, great-expectations, evidently

Basket value x (EUR)	Reference ECDF (10, 20, 30, 40, 50)	Current ECDF (30, 40, 50, 60, 70)	Gap
10	0.2	0.0	0.2
20	0.4	0.0	0.4
30	0.6	0.2	0.4
40	0.8	0.4	0.4
50	1.0	0.6	0.4
60	1.0	0.8	0.2
70	1.0	1.0	0.0

Basket value x (EUR)	Reference ECDF (10, 20, 30, 40, 50)	Current ECDF (30, 40, 50, 60, 70)	Gap
10	0.2	0.0	0.2
20	0.4	0.0	0.4
30	0.6	0.2	0.4
40	0.8	0.4	0.4
50	1.0	0.6	0.4
60	1.0	0.8	0.2
70	1.0	1.0	0.0

1 Recall · 5 min

2 Concept · 10 min