Core day 18 of 60 · Week 4 · Part I · evaluation depth

Error Analysis and Debugging

A candidate model improves overall MAE on the D-14 bookings forecast from 7.1 to 6.0 seats, and the release note says "better across the network". Where do you look to test that claim, how do you decide which failure to fix first, and what do you check when RMSE improves by 15% and the business cost still rises?

By the end: Given a model's errors you slice them, name the residual pattern, rank the slices by volume x error x unit cost, decompose the change between two versions, and state one root-cause hypothesis with the test that could refute it.

At a glance

An aggregate can improve while a costly slice gets worse; slice, read the pattern, rank by total cost, decompose the change and test one cause.

Slice first
Pre-chosen slices from decision-time information, each with count, bias and error.
Read the shape
Bias, a funnel, clusters in time or a perfect slice each point to a different cause.
Rank by total cost
Volume x error x unit cost, not error rate: a loud small slice is rarely first.
Decompose sums
Waterfall steps add up only for sums (seats, euros, squared error), never RMSE.
One hypothesis
Name the cause, the observation that would refute it and the cheapest check.

Recall · 5 min

Without notes: (1) which metric would you report for a volume-weighted forecast error, and when is MAPE undefined (Day 16)? (2) what is the difference between permutation importance and an intervention effect (Day 17)? (3) which residual property tells you a forecast is biased (Day 11)? Then clear your due review cards.

Open due reviews

Concept · 10 min

An aggregate metric is a volume-weighted average over slices. It can improve while the slice with the highest stakes gets worse. Error analysis replaces one number with a map: where the errors are, what shape they have and why they might be there. Like a building-wide smoke alarm, the aggregate says there is a fire, not which room.

The aggregate improves while one slice gets worse, and ranking by error rate points at the wrong slice

Predict first

A slice has 30 rows and the highest MAE of any slice. Another has 450 rows and a lower MAE, a systematic bias. Which does ranking by volume x error put first?

Text description of the figure

One seeded forecast table (nine routes, seven weekdays, three booking horizons, four weeks) with a switch for the slicing dimension (route, market, weekday, horizon) and the metric (WAPE, MAE, bias). A bar chart shows the metric per slice against the overall line, and a table ranks slices by volume times error beside their rank by error rate. It compares one forecaster with the actuals, not two versions, and has no waterfall; the waterfall is the extra visual below.

Five steps from one number to a map Sequence

Slice
By what the decision varies on: region, product line, channel, season, horizon, data source.
Read the pattern
Bias, spread, clusters in time, jumps at one horizon, suspiciously perfect slices.
Rank
Volume x error x unit cost, with count and bias beside each slice.
Decompose
A waterfall of additive sums shows which slice moved the metric between versions.
Test one hypothesis
Suspected cause, refuting observation, cheapest check. Then change the model.

Debug in order of cost: pipeline, leak, slices, then the model Diagram

Metric looks offa move, a gap, a complaint
Pipeline changed?schema or code
- no: to Too good anywhere?
yes
Fix the pipelineskew, stale table, broken join
Too good anywhere?a perfect slice
- no: to Slice and read residuals
yes
Hunt the leaktimestamps against the cutoff
Slice and read residualscount, bias, error, cost
One hypothesiswith a refuting test
confirmed
Model changefeatures, objective, capacity

Pipeline means schema, nulls, feature code or an upstream table; too good means a perfect slice or one dominant feature. Cheap, common causes first; a refuted hypothesis sends you back to the slices for the next one. Changing the model is the last step, not the first.

Two rankings of the same slices (candidate B, application script) Comparison

Criterion	new-route	intl-long
Rows	30	450
MAE (seats)	18.8, the highest	16.0
Bias (seats)	-1.3, mostly noise	-15.3, systematic under-forecast
Total error (seats)	563	7,191, the highest
Rank by rate	1st	2nd
Rank by volume x error	4th	1st

Error rate finds defects; total error in business units sets priority. Bias and count tell a defect from noise.

Waterfall: seats of error, A to B, slice by slice Bridge

Model A, all 3,000 departures21,417 seats
domestic−3,816 seats
intl-short−2,468 seats
intl-longabout EUR 213,000 worse at EUR 80 per seat+2,657 seats
new-route+58.9 seats
Model B, all departures17,848 seats

Output of the application script (total absolute error in seats). Both versions are scored on the same rows, so there is no mix term: the steps are pure rate changes and add up exactly to the aggregate change, -3,569 seats. The aggregate improvement hides one slice that pays for it.

Key terms: Error slice, Residual pattern, Metric waterfall, Failure taxonomy, Training-serving skew, Data drift (covariate shift) and Data leakage. The five steps, the ranking rule and the debugging order are in the pictures above; residual patterns, drift, leakage and learning curves are in the optional depth below.

Assumptions

Slices are defined from information available at decision time, not from the outcome; slicing on the target manufactures patterns.
The slice has enough rows for its error to be a signal; with a few dozen rows the error is mostly noise.
Unit costs are stated, and the same costs apply to both versions being compared.

Application and workplace decision · 10 min

Reuse the D-14 bookings forecast. The script simulates 3,000 departures in four slices and compares:

A, the incumbent: unbiased;
B, the candidate: less noise, but it has seen too little long-haul data.

It prints the aggregate MAE, then per slice the count, MAE of both versions, B's bias, seats of error and the waterfall step, and checks that the steps add up. Before running it, predict which slice B hurts and whether the aggregate will show it.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
import pandas as pd

rng = np.random.default_rng(18)
n = {"domestic": 1500, "intl-short": 1020, "intl-long": 450, "new-route": 30}
df = pd.DataFrame({"slice": np.repeat(list(n), list(n.values()))})
level = df["slice"].map({"domestic": 150, "intl-short": 155, "intl-long": 250, "new-route": 90})
df["y"] = rng.poisson(level).astype(float)                           # final bookings (synthetic)
# A = incumbent (unbiased). B = candidate: less noise, but 16 seats too low on long-haul it saw too little of
sd_a = df["slice"].map({"domestic": 8, "intl-short": 8, "intl-long": 12, "new-route": 20})
sd_b = df["slice"].map({"domestic": 5, "intl-short": 5, "intl-long": 10, "new-route": 25})
df["err_a"] = rng.normal(0, sd_a)                                    # forecast - actual, in seats
df["err_b"] = rng.normal(0, sd_b) + np.where(df["slice"] == "intl-long", -16.0, 0.0)

print(f"all departures: MAE A {df.err_a.abs().mean():.2f} -> B {df.err_b.abs().mean():.2f} seats")
g = df.groupby("slice").agg(n=("y", "size"), mae_a=("err_a", lambda s: s.abs().mean()),
                            mae_b=("err_b", lambda s: s.abs().mean()), bias_b=("err_b", "mean"))
g["seats_b"] = g["n"] * g["mae_b"]                                   # volume x error
g["step_seats"] = g["n"] * (g["mae_b"] - g["mae_a"])                 # waterfall step: B minus A
g["step_eur_k"] = g["step_seats"] * 80 / 1000                        # EUR 80 per seat, assumed linear
print(g.sort_values("seats_b", ascending=False).round(1).to_string())
print(f"highest MAE: {g.mae_b.idxmax()} | highest volume x error: {g.seats_b.idxmax()}")
print(f"sum of steps {g.step_seats.sum():,.0f} seats = total change {len(df) * (df.err_b.abs().mean() - df.err_a.abs().mean()):,.0f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 36 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

Bias-variance algebra: what an expected error is made of7 minThe decomposition written out, derived in seven lines, computed by hand on four refits, and what each term tells you to do.

Bias-variance decomposition: three sources of expected squared error Formula

\mathbb{E}\big[(Y - \hat f(x_0))^2\big] = \underbrace{\sigma^2}_{\text{noise}} + \underbrace{\big(f(x_0) - \mathbb{E}\hat f(x_0)\big)^2}_{\text{bias}^2} + \underbrace{\mathrm{Var}\,\hat f(x_0)}_{\text{variance}}

$Y = f(x_0) + \varepsilon$: the outcome at input x0: a true mean f(x0) plus noise with mean 0 and variance sigma squared
$\hat f(x_0)$: the model's prediction at x0; random because the training sample is random
$\mathbb{E}\hat f(x_0)$: the average prediction over many training samples of the same size
$\sigma^2$: irreducible noise: no model fitted to these inputs can go below it
$\text{bias}^2$: how far the average prediction sits from the truth: too simple a model, a missing feature, a shifted slice
$\mathrm{Var}\,\hat f(x_0)$: how much the prediction moves when the training sample changes: too flexible a model, too few rows

The expectation is over both the noise in Y and the training sample behind the model. Learning curves (last chapter) estimate the three terms from data.

Derivation: add and subtract, then the cross terms vanish Sequence

Write the error as three gaps
Y - f-hat = (Y - f) + (f - E f-hat) + (E f-hat - f-hat), all at x0. Square it and take the expectation.
Noise term
E(Y - f)^2 = E(epsilon^2) = sigma squared.
Bias term
(f - E f-hat)^2 is a constant: it does not depend on the noise or on the training sample.
Variance term
E(E f-hat - f-hat)^2 = Var f-hat, by the definition of a variance.
Cross term with noise
2 E[epsilon (f - f-hat)] = 0: the new outcome's noise is independent of the training sample, and E epsilon = 0.
Cross term bias x variance
2 (f - E f-hat) E(E f-hat - f-hat) = 0, because f-hat averages to E f-hat.
Result
Expected squared error = sigma squared + bias squared + variance. Averaged over rows, the same split holds for the test MSE.

Bias and variance by hand: one input, four training samples Table

Training sample	Prediction at x0	Gap to truth (12)	Squared deviation from mean prediction
1	9	-3	1
2	11	-1	1
3	12	0	4
4	8	-4	4
Average	10	bias -2, so bias squared 4	variance 10 / 4 = 2.5

Predicted weeks to failure for one pump design at a factory, refitted on four samples (illustrative); true mean 12, noise variance 4. Expected squared error = 4 + 4 + 2.5 = 10.5. Directly: 4 + mean of 9, 1, 0, 16 = 10.5. A simulation with 400,000 draws gave 10.51.

The pump example as a bridge: noise, then bias, then variance Bridge

Noise sigma squared4 weeks squared
Bias squaredaverage prediction 10 against truth 12+4 weeks squared
Variancespread of the four refits+2.5 weeks squared
Expected squared error10.5 weeks squared

The three terms from the table above, stacked. The noise bar is the floor no model reaches; only the bias and variance bars respond to model choices.

Each error term has its own lever Hierarchy

Expected MSE at x0noise + bias squared + variance
- irreducibleNoise sigma squareda floor; read error against it
- too simpleBias squaredmore capacity, the missing feature
- too flexibleVariancemore rows, shrinkage, bagging

The split holds per input x0: a model can be biased in one slice and fine elsewhere, which is what slicing finds. Variance also falls with averaging (bagging, Day 12). For absolute error or log loss the three-term split does not hold exactly, but the diagnosis still separates a systematic offset from spread.

Slice algebra: bias and spread, mix and rate, a waterfall8 minWhy additive metrics decompose and RMSE does not, a slice's MSE split into bias and spread, and a period change split into mix and rate.

Pick the decomposition before you draw the waterfall Diagram

A metric changedbetween versions or periods
A sum or a mean?seats, MSE: yes
- yes: to Same rows in both?
no
Split the partsnumerator and denominator; ratio once, last
then
Same rows in both?versions: yes
- no: to Mix plus rate
yes
Waterfall of rate stepssteps add up exactly
Mix plus ratetwo periods: check the mix first

RMSE, WAPE and other ratios are reported once, at the end, from the decomposed sums.

Why squared error decomposes and RMSE does not Formula

\Delta\mathrm{MSE} = \sum_{s} \frac{n_s}{n}\left(\mathrm{MSE}'_s - \mathrm{MSE}_s\right), \qquad \mathrm{RMSE} = \sqrt{\mathrm{MSE}}

$n_s / n$: share of rows in slice s; the weights of a volume-weighted average
$\mathrm{MSE}'_s - \mathrm{MSE}_s$: the slice's change; its weighted terms add up exactly
$\sqrt{\cdot}$: the square root is not additive: decompose MSE (or sums), report RMSE once

Same for ratios such as WAPE or MAPE-like rates; decompose numerator and denominator, then divide.

One slice's MSE splits into bias squared plus spread (by hand) Table

Slice	Errors (forecast minus actual)	Mean error	Spread (variance)	MSE = mean squared + spread
Analyzer 1	-18, -14, -16, -12	-15	5	225 + 5 = 230
Analyzer 2	+6, -6, +4, -4	0	26	0 + 26 = 26

Hospital lab QC on a control sample (mg/dL, illustrative). Analyzer 1 is off by a constant: recalibrate it, one shift fixes 98% of its MSE. Analyzer 2 is unbiased but noisy: a shift does nothing; replicates or maintenance do. This is the empirical bias-variance split, and the reason to show bias beside every slice.

Between periods, an aggregate moves by mix and by rate Formula

\bar m' - \bar m = \underbrace{\sum_s (w'_s - w_s)\, m_s}_{\text{mix}} + \underbrace{\sum_s w'_s\,(m'_s - m_s)}_{\text{rate}}, \qquad \bar m = \sum_s w_s\, m_s

$w_s, w'_s$: slice s's share of rows before and after (for two versions on the same rows, w' = w and the mix term is 0)
$m_s, m'_s$: the slice's metric before and after, for an additive metric such as MAE, MSE or cost per row
$\text{mix}$: the change from rows moving between slices of different difficulty, at the old rates
$\text{rate}$: the change from the slices themselves getting better or worse, at the new mix

The split is exact but not unique; using the old mix in the rate term gives a different split. Averaging both orders is the symmetric (Shapley) choice from Day 17.

Both slices improve, the aggregate worsens: mix and rate by hand Table

Slice	Period 1: rows, MAE	Period 2: rows, MAE	Mix term (w' - w) m	Rate term w' (m' - m)
Urban	800, 10 min	500, 9 min	(0.5 - 0.8) x 10 = -3.0	0.5 x (-1) = -0.5
Rural	200, 30 min	500, 28 min	(0.5 - 0.2) x 30 = +9.0	0.5 x (-2) = -1.0
All deliveries	MAE 14.0	MAE 18.5	+6.0	-1.5

Parcel-delivery ETA errors (illustrative). Total +4.5 = +6.0 mix - 1.5 rate: the model got better in both slices and the business shifted toward rural routes (Simpson's paradox, not a regression). The other order gives +5.7 and -1.2.

Waterfall: seats of error, A to B, slice by slice Bridge

Model A, all 3,000 departures21,417 seats
domestic−3,816 seats
intl-short−2,468 seats
intl-longabout EUR 213,000 worse at EUR 80 per seat+2,657 seats
new-route+58.9 seats
Model B, all departures17,848 seats

Slicing without manufacturing patterns4 minWhich dimensions to slice by, why small slices look bad by chance, and why slicing on the outcome invents a bias that is not there.

What to slice by, and what each slice can reveal Table

Slice by	Known at decision time?	Can reveal
Region, store, product line, device	Yes	Coverage gaps, definition changes, segment bias
Season, holiday, event	Yes	Missing calendar features, rare regimes
Lead time or horizon	Yes	Features that arrive late, leakage at short horizons
Forecast level (deciles of the prediction)	Yes	Calibration and level-dependent bias
Data source, missing-feature flags	Yes	Pipeline and coverage problems
Actual outcome (deciles of y)	No	Nothing: it manufactures opposite biases at the ends

Small slices look biased by chance; a real defect stands outside the funnel Illustration

Synthetic errors: 100 unbiased slices of 20 to 2,000 rows (log-uniform sizes, row errors normal with sd 8 seats) and one slice of 450 rows with a planted bias of -4 seats. Points beyond the axis are drawn at its edge.

6 of 100 healthy slices cross the 95% band by chance (2 of them under 100 rows). A 54-row healthy slice shows a mean error of 2.4 seats; the real defect, 450 rows at -4.3, sits far outside the band where the band is narrow.

Pre-register the slice list from the decision, require a minimum count and show the interval: a few dozen rows is mostly noise, and at a 5% level about 5 of 100 healthy slices cross the line. Treat unplanned slices that cross as hypotheses for a later period.

The same unbiased forecasts, sliced two ways Illustration

Lowest decile: forecast 7.9 units above the actual; highest: 7.3 below. A pattern made by the slicing, not the model.

Every decile within 0.5 units of zero: the same forecasts, honestly unbiased.

Synthetic data: 4,000 rows; each forecast equals the true expected value and the actual adds noise with sd 8 units. Overall mean error 0.08 units. Bars: mean error per decile (400 rows), whiskers ±2 standard errors.

Grouping rows by what happened selects the rows where noise pushed the actual up or down. Grouping the same rows by their forecast shows no bias.

Why slicing on the outcome manufactures bias Formula

\mathbb{E}\left[\hat y - y \mid y\right] = -\frac{\sigma_{\varepsilon}^{2}}{\sigma_{\mu}^{2} + \sigma_{\varepsilon}^{2}}\,\left(y - \bar y\right)

$\hat y = \mu$: a forecast equal to the expected value: unbiased for every row
$y = \mu + \varepsilon$: the actual, with noise variance sigma_epsilon squared
$\sigma_{\mu}^{2}$: spread of the expected values across rows (the signal)
$y - \bar y$: how far a row's actual is from the average; the further, the larger the apparent bias

Normal-theory identity (regression to the mean). In the demo, sd 12 for the signal and 8 for the noise give a factor of 64 / 208, about 0.31.

Residual patterns and their first suspects3 minEight patterns, what each usually means, and the cheapest test that could refute it.

Four residual shapes and the number that flags each Illustration

Rows of one segment (red) average -9.0 units, the rest -0.1. First suspect: coverage or a definition change for that slice.

Honest model: MAE 2.4 at 2 days, 11.9 at 30. Leaky model (red): 2.4 and 2.4. Flat error means a feature known only later: check timestamps against the cutoff.

Lag-1 autocorrelation 0.80 (about 0 if residuals were white). First suspect: missed seasonality, events or a lagged feature.

Mean error 0.2 before day 100, 9.1 after. First suspect: a pipeline change or a structural break on that date.

Synthetic residuals (forecast minus actual, units; panel 2 shows MAE over 60 rows per horizon), 160 rows per panel, each generated with exactly one planted defect; the numbers in the notes are computed from the plotted points.

Each panel has exactly one planted defect. Good residuals have zero mean and no structure in time; structure means usable information was left out (Day 11). The first suspect follows from the shape, the number says how strong it is.

Residual pattern, first suspect, first test Table

Pattern	First suspect	First test
Mean error away from zero in one slice	Little training data there, or its definition changed	Compare train and test counts and the target definition for the slice
Spread grows with level (funnel)	Errors are relative, not absolute	Plot error against the forecast; try a log or ratio target
Residuals cluster in time order	Missed seasonality, events or a break	Plot residuals by date; check their autocorrelation
Level shift after a date	Structural break or a pipeline change	Compare feature distributions before and after the date
Error flat across horizon when it should grow	A feature known only later	Check feature timestamps against the cutoff (Day 2)
Perfect score on a slice	Leakage	Remove the slice's strongest feature and rescore
False positives cluster in a slice	Threshold or prevalence changed there	Precision and counts at the fixed threshold by slice
Confident wrong predictions (classifier)	Label errors or a leak	Fix the threshold from costs, then read the most confident errors first

When RMSE and cost disagree3 minThe lab's version trades a rare tail for a bias in the bulk; slice by direction to see it.

Hands-on lab: candidate versus incumbent Chart

RMSE incumbent9.46
RMSE candidate-15%8.05
MAE incumbent6.98
MAE candidate6.49

Seats. Output of the hands-on lab script (4,000 synthetic departures). Both error metrics improve.

Where the RMSE gain and the cost rise come from (lab output) Table

Slice	Rows	Change in squared-error sum	Change in cost
holiday spike	67	-95,910	EUR -46,718
regular	3,933	-3,072	EUR +400,128
all departures	4,000	-98,982	cost per departure EUR 366 -> 455

The tail fix buys almost all the squared-error gain; a 4-seat upward bias on regular days at EUR 80 per empty seat pays for it. Remove the bias, or train an objective that matches the asymmetric cost (a quantile loss at the cost ratio, Day 16), then compare by slice again.

A failure taxonomy2 minTag each failure as world, data, pipeline, model or decision so recurring causes become visible.

A failure taxonomy (tag every incident with one layer) Stack

WorldNew competitor, tariff or policy change, pandemic, product relaunch: the relationship itself moved.
DataCoverage gaps, label noise, definition changes, delayed or censored labels.
PipelineTraining-serving skew, stale tables, broken joins, time-zone and unit bugs.
ModelMissing feature, wrong objective, too little or too much capacity, poor calibration.
DecisionWrong threshold or cost assumption: the model is fine, the action rule is not.

Tagged incidents turn one-off fixes into recurring causes you can prevent.

Training-serving skew, data drift and concept drift4 minFour ways a model that validated well goes wrong live, each with its own symptom and first check.

Skew and drift are different failures Comparison

Criterion	Training-serving skew	Data drift	Concept drift	Label shift
What changes	Same input computed differently online and offline	Input distribution P(X)	Relationship P(y given X)	Class or target mix P(y)
Typical symptom	Offline score fine, live error worse from day one	Inputs leave the training range; error may or may not rise	Same inputs, different outcomes; residual bias appears	Precision moves at a fixed threshold; AUC can hold
First check	Log served features; compare with offline values per row	Compare feature distributions with the training window	Residuals over time once labels arrive	Prevalence by period and slice
Usual fix	One feature definition shared by training and serving	Range guards, retrain on recent data if error rises	Retrain, new features, or a regime indicator	Prior correction or a threshold review (Day 13)

Skew is a bug and shows from the first day; drift builds up over time. Drift in inputs alone is not proof of harm, and concept drift can occur with no input drift at all: it needs labels to see, so input monitoring alone can miss it (Day 19).

Leakage symptoms2 minFive signs that the model saw information it will not have at decision time.

Leakage symptoms (a leak until shown otherwise) Overview

Too good to be true
A jump far beyond the baseline and earlier attempts with no new information.
One dominant feature
A single column carries most of the importance; check when it is recorded.
Perfect slice
Zero error on a segment, often where a post-outcome field is filled.
Flat across horizon
Error should grow with lead time; flat error means information from the future.
Collapse in production
Live error far above validation from day one, without input drift.

Learning curves: bias or variance?advanced3 minPlot training and validation error against training size to decide between more data, more capacity or more regularization.

Learning curves from a real sweep: bias, balance, variance Illustration

Both curves settle near 0.38, well above the noise floor 0.25: more rows will not help, more capacity will.

Gap at 640 rows: 0.09. Validation ends at 0.30, close to the floor.

Gap 0.49 at 20 rows, 0.26 at 640: more rows help, and so does regularization (shallower trees, larger leaves).

training error (MSE)
validation error on 400 fresh rows

Synthetic data: a smooth curve plus noise (sd 0.5, so the best possible MSE is 0.25). Each point averages three refits of the app's regression tree on fresh training samples.

The same data, three tree depths, training sets from 20 to 640 rows. Where the curves meet, and how far above the noise floor (estimate it from repeat measurements or a strong baseline, never use zero), tells you whether rows, capacity or regularization will help.

Reading a learning curve Table

What you see	Diagnosis	What helps
Training and validation close, both high	High bias (underfitting)	More features, more capacity, less regularization
Training low, validation far above	High variance (overfitting)	More rows, regularization, simpler model, early stopping
Validation still falling at the largest size	Data-limited	Collect more rows before tuning hard
Validation below training	Check the split	Leakage, duplicates across folds, or an easier validation period
Validation stalls above the floor as rows grow	A tree grown to one row per leaf acts like a nearest-neighbour rule	Cap depth or raise the leaf size first

scikit-learn 1.9 learning_curve defaults to 5 training sizes from 10% to 100%, 5-fold CV and shuffle=False; for time-ordered data pass cv=TimeSeriesSplit.

Memory anchor

Slice, read the shape, rank by volume x error x cost, decompose the change, test one hypothesis.

Useful for

Reviewing a model release or an A/B result for segments that regressed under an improving aggregate.
Turning a vague "the forecast feels off" into a ranked list of hypotheses with first tests.
Explaining a metric change between two versions in business units.

Not suitable for

Proving a cause; a slice shows where errors are, and only a controlled change or experiment tests why.
Very small samples or very many slices without correction; some slices will look bad by chance.

Traps

Slicing after seeing the outcome

Slices defined by the target or by errors found only after looking at the results manufacture patterns, and a slice chosen from 200 candidates will look bad by chance.

How to avoid it

Pre-register the slice list from the decision, require counts and intervals, and treat other slices as hypotheses.

Fixing the loudest slice

The highest error rate gets the engineering time, while a larger slice with a moderate bias carries the cost.

How to avoid it

Rank by volume x error x unit cost and use bias and counts to separate defects from noise.

Senior reasoning

A senior distrusts a single aggregate, asks for error by the slices the decision depends on, and prices the differences in the unit the business uses. They decompose the change between versions into additive steps, separate bias from variance, rank work by total cost, and write each hypothesis with the test that could prove it wrong before changing the model.

Interview question

"Your new model has a better overall metric, but a stakeholder says it is worse for their region. What do you do?" How would you structure the answer?

Outline your answer first.

Recall cards scheduled after completion

What is the right order for ranking error slices, and why is error rate alone misleading?
Why can RMSE improve while cost rises, and what is the first check?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Rules of Machine Learning: Best Practices for ML Engineering — Martin Zinkevich (Google)core
Rule 26 (look for patterns in the measured errors) and Rule 27 (turn observed undesirable behavior into measurable numbers) support slicing and quantifying failures. (Rules 26 and 27)
Forecasting: Principles and Practice (3rd edition) — Rob J Hyndman, George Athanasopoulos, 2021core
Residual diagnostics, section 5.4; good residuals are uncorrelated and have zero mean, so correlated or biased residuals mean usable information was left out. (5.4 Residual diagnostics)
The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction — Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, D. Sculley, 2017optional
The rubric lists tests on model quality by data slice as part of production readiness, which a release note needs.
Validation curves and learning curves (scikit-learn User Guide 3.5) — scikit-learn developersoptional
Defines training and validation learning curves and how to read underfitting and overfitting from them.
Metrics and scoring: quantifying the quality of predictions (scikit-learn User Guide 3.4) — scikit-learn developersoptional
Confusion-matrix based metrics used for false-positive and false-negative diagnosis by slice. (3.4 Metrics and scoring)

Optional sections

Deep dive · 10 min

Why RMSE is not additive. For $n$ rows in slices $s$ , $\text{MSE}=\sum_s \frac{n_s}{n}\text{MSE}_s$ , so the difference between two versions decomposes exactly into $\sum_s \frac{n_s}{n}(\text{MSE}'_s-\text{MSE}_s)$ . RMSE is the square root of that sum. Its change cannot be split into per-slice pieces that add up, so decompose MSE (or squared-error sums), then report RMSE once. The same holds for ratios such as WAPE: decompose numerator and denominator.

Classifier debugging. Fix the threshold from costs (Day 13), then slice precision, recall and the count of false positives and false negatives. Read the most confident wrong predictions first; they are usually label errors, a missing feature or leakage reversed. Compare score distributions by slice before touching the threshold.

Residual patterns in time. Plot residuals in time order and by horizon. A level shift after a date is a break or a pipeline change, a seasonal pattern means missing calendar features, and growth with horizon is expected. Error that is flat in horizon when it should grow can mean a feature known only later (Day 2).

A small failure taxonomy. Data (coverage, definition, label noise), model (capacity, objective, missing feature), pipeline (training-serving skew, stale tables), and world (new competitor, policy). Tag each failure with one so recurring causes become visible.

Hands-on lab: RMSE improves, cost rises - find the slice and the direction · 15 min

Run the script. It simulates 4,000 departures. The incumbent misses rare holiday spikes by about 37 seats (an under-forecast); the candidate catches them but runs about 4 seats high everywhere. It prints RMSE, MAE and cost, then for each slice the change in squared error and in cost. Answer: (a) which slice carries the RMSE gain and which carries the cost rise; (b) why do the squared-error steps still sum to the total change in sum of squares (and RMSE itself does not); (c) which objective would you train so the model stops trading tail for bulk?

import numpy as np
import pandas as pd

rng = np.random.default_rng(180)
n = 4000
spike = rng.random(n) < 0.02                       # holiday-week departures where demand jumps (synthetic)
e_old = rng.normal(0, 8, n) - np.where(spike, 37, 0)   # incumbent: fine on regular days, misses spikes by ~37 seats
e_new = rng.normal(4, 7, n)                         # candidate: catches spikes, but runs 4 seats high everywhere
cost = lambda e: np.where(e > 0, 80.0, 30.0) * np.abs(e)   # EUR: empty seat 80, diluted fare 30 (assumed)
df = pd.DataFrame({"slice": np.where(spike, "holiday spike", "regular"), "e_old": e_old, "e_new": e_new})
rm = lambda e: np.sqrt((e ** 2).mean())
print(f"RMSE {rm(e_old):.2f} -> {rm(e_new):.2f} ({100 * (rm(e_new) / rm(e_old) - 1):+.0f}%)")
print(f"MAE  {np.abs(e_old).mean():.2f} -> {np.abs(e_new).mean():.2f}")
print(f"cost per departure EUR {cost(e_old).mean():.0f} -> {cost(e_new).mean():.0f}")
for name, g in df.groupby("slice"):
    print(f"{name:<14} n={len(g):4d}  dMSE x n = {(g.e_new**2 - g.e_old**2).sum():>10,.0f}  "
          f"dCost = EUR {(cost(g.e_new) - cost(g.e_old)).sum():>9,.0f}")

Interview practice · 10 min

A bank's card-fraud model has a stable AUC in production, but the fraud-operations team says the alerts "feel worse this month". How do you investigate?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-11 · depends on numpy, pandas