Core day 16 of 60 · Week 4 · Part I · evaluation depth

Metrics and Business Objectives

A candidate model cuts MAPE on the D-14 bookings forecast from 14.9% to 12.2%, and a slide calls it a win. Before anyone ships it: what does MAPE weight, when does it become undefined or misleading, and what does the change cost in seats and euros?

By the end: For a forecast or classifier you choose the metric that matches the decision, write its exact formula, say when it is undefined or misleading, and convert the error into seats or euros per decision.

At a glance

A metric decides which mistakes count: choose it from the decision's cost, know when it breaks, and translate every change into units and money.

What it estimates
MAE aims at the median, RMSE at the mean, pinball loss at a quantile; MAPE pulls forecasts low.
Volume and scale
MAPE averages percentages equally, WAPE weights by volume, MASE compares with a naive forecast.
Undefined is not 0
A zero actual breaks MAPE; report "undefined" with the reason, never a zero.
Cost decides
Business metric = error x volume x unit cost; asymmetric costs pick the quantile.
Targets get gamed
Goodhart: pair each target with a guardrail and an interval, and re-check it against cost.

Recall · 5 min

Without notes: (1) name the Day 2 framing-card items and say which one gives you the metric unit and the price of one error; (2) say what a threshold should be derived from (Day 13); (3) say what Day 15 does to a score that was used to choose among 50 tuned variants. Then clear your due review cards.

Open due reviews

Concept · 10 min

A metric is an agreement about what counts as a mistake. A 4-unit miss is 33% at a 12-unit corner store and 2% at a 180-unit flagship: MAE calls them equal, MAPE calls the first fifteen times worse, and the replenishment planner sees about four units each. Match the metric to the decision and know when it stops answering.

The same forecast errors rank models differently under MAPE, WAPE, RMSE and cost

Predict first

One actual becomes 0 while its forecast stays at 12 and all other rows are unchanged. Which metric can no longer be computed as defined?

Text description of the figure

A table of actuals and forecasts you can edit, with every metric recomputed beside it. Setting an actual to zero turns MAPE into "undefined" while WAPE and MAE stay defined; adding a spike moves RMSE far more than MAE; scaling all values leaves percentage metrics unchanged and rescales MAE and RMSE.

One sample, three metrics, three different best forecasts Illustration

Median 8.0, mean 10.3: a long right tail pulls the mean up.

Straight pieces with a kink at every actual: lowest at the median, where as many rows sit above as below.

Smooth bowl: lowest at the mean, so the tail drags the best forecast up.

Lowest below the median: each row weighs 1 / actual, so small actuals pull the best forecast down.

Synthetic data: 61 seeded right-skewed actuals (log-normal, rounded, at least 1 unit). Each curve is computed by sweeping the constant from 0 to 40 units in quarter-unit steps. Actuals above 40 units (1) lie beyond the axis but count in every curve.

Sweep a single constant forecast over skewed demand. MAE bottoms out at the median, RMSE at the mean, MAPE below both. Choosing the metric chooses the number you will forecast.

Choosing the metric from the decision Diagram

Decisionwho acts on the forecast
Price each errorper unit, both directions
Same price both ways?
- no: to Pinball loss
yes
MAE or RMSEmedian or mean forecast
- to Mixed sizes or units?
Pinball lossat q = cu / (cu + co)
Mixed sizes or units?
- no: to Euros by segment
yes
Add WAPE or MASEto aggregate across series
Euros by segmentthe number a decision maker reads

cu and co are the costs of one unit under- and over-forecast. MAPE stays a diagnostic, not the choice.

Four forecast metrics side by side Comparison

Criterion	MAE	RMSE	MAPE	WAPE
Best constant forecast	The median	The mean	Below the median (weights 1 / actual)	The median (MAE over a fixed total)
Unit	Units	Units	Percent	Percent of volume
Each row weighs by	Its error in units	Its squared error: big misses dominate	1 / actual: small rows dominate	Its error in units, over total volume
Zero actuals	Defined	Defined	Undefined	Undefined only if all are zero
Use for	Symmetric cost per unit	Costs that grow faster than the miss; mean forecasts	A diagnostic on positive series of similar size	A headline across series of different size

B wins on small departures and loses where the money is Chart

A, all departures628 EUR per departure
B, all departuresMAPE better, euros worse888 EUR per departure
A, small departures113 EUR per departure
B, small departuresB wins where seats are few43 EUR per departure
A, large departures1,249 EUR per departure
B, large departuresand loses where the money is2,224 EUR per departure

Computed by the application script (synthetic, EUR 80 per seat). MAPE fell from 14.9% to 12.2% while cost rose from EUR 628 to EUR 888: a percentage averaged over unequal volumes lets many cheap gains outvote a few expensive losses. Convert every change into error x volume x unit cost before calling it a win.

Key terms: MAPE, WAPE, MASE, pinball loss, log loss, PR-AUC, critical ratio, Goodhart's law and block bootstrap.
Undefined means undefined: report it with the reason, never as 0.
Business metric = error x volume x unit cost; asymmetric costs make the best forecast a quantile.
Formulas, what each metric estimates, aggregation and uncertainty are in the pictures above and the optional technical depth below.

Assumptions

The evaluation rows resemble deployment decisions; a metric computed on tuned or leaked data misleads before any formula is chosen (Day 15).
Unit costs are stated and stable enough to rank models; EUR 80 per seat is an assumption to replace with finance's number.
Errors are comparable across rows; with different scales or meanings of zero, use scaled or volume-weighted metrics.

Application and workplace decision · 10 min

Day 2 card: D-14 bookings forecast, about EUR 80 per seat of error, assumed linear.
Data: 300 synthetic departures in three volume classes (about 8, 40 and 160 seats still to be picked up).
Models: A is the current forecaster; B is a candidate with less noise on small departures and more on large ones.
The script prints MAPE, WAPE, RMSE and euros per departure, overall and by class, then adds one cancelled departure.
Predict first: does the MAPE gain carry over to euros?

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np

rng = np.random.default_rng(16)
seg = np.repeat(["small", "mid", "large"], 100)          # 300 departures in three volume classes
mu = np.repeat([8.0, 40.0, 160.0], 100)                  # synthetic bookings still to be picked up
y = rng.poisson(mu).astype(float)
sd = {"A": [0.22, 0.20, 0.12], "B": [0.08, 0.16, 0.20]}  # assumed relative noise per class: A current, B candidate
cls = np.repeat([0, 1, 2], 100)
fc = {k: np.maximum(0, y * (1 + rng.normal(0, np.array(v)[cls]))) for k, v in sd.items()}

def metrics(y, f, cost=80.0):  # an undefined metric stays None - it is never reported as 0
    e = np.abs(y - f)
    return {"MAPE%": None if (y == 0).any() else 100 * np.mean(e / y),
            "WAPE%": None if y.sum() == 0 else 100 * e.sum() / y.sum(),
            "RMSE": np.sqrt(np.mean(e ** 2)), "EUR/dep": cost * e.mean()}

def show(label, m):
    print(f"{label:<26}", "  ".join(f"{k} {'undefined' if v is None else format(v, '.1f')}" for k, v in m.items()))

for k in "AB":
    show(f"model {k}, all", metrics(y, fc[k]))
for name in ["small", "large"]:
    for k in "AB":
        m = seg == name
        show(f"  model {k}, {name}", metrics(y[m], fc[k][m]))
show("model B + 1 zero actual", metrics(np.append(y, 0.0), np.append(fc["B"], 3.0)))

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 45 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The object: a loss, its risk, and the statistic it rewards5 minWhat a metric estimates, the exact definition of every metric in this lesson, and the assumptions as math rather than slogans.

A metric is an estimate of a risk Formula

R(f) = \mathbb{E}\,L\big(Y, f(X)\big) \;\approx\; \frac{1}{n}\sum_{t=1}^{n} L(y_t, \hat y_t), \qquad f^*(x) = \arg\min_c \mathbb{E}\big[L(Y, c) \mid X = x\big]

$L(y, \hat y)$: the loss: the price of forecast y-hat when the outcome is y; |e| for MAE, e squared for MSE, L_q for pinball
$R(f)$: the risk: the expected loss of forecaster f over the deployment distribution of (X, Y)
$\frac{1}{n}\sum_t$: the metric you report: the average loss on n evaluation rows, an estimate of R(f)
$f^*(x)$: the best possible forecast: the statistic of Y given x that the loss rewards (mean, median or a quantile)

Assumptions, written as math: evaluation rows are draws from the deployment distribution (Day 15) and the loss adds up over rows, so one average is meaningful. Ratio metrics (MAPE's weights, WAPE, R-squared) bend the second.

Most metrics are not plain averages of a row loss Hierarchy

A reported metriccomputed on the evaluation rows
- plain meanMAE, MSE, pinballand log loss: estimate the risk
- re-weightedMAPErow weights 1 / y: small actuals dominate
- ratio of sumsWAPEsum of |e| over sum of y: volume carries it
- vs a baselineMASE, R-squarednaive MAE; test-window variance

A loss elicits the statistic that minimizes its expected value: median, mean or quantile. Change the rows (a calmer window, dropped zeros) and any metric estimates a different risk. Rows from neighbouring days or one store keep the average unbiased but carry less information than n independent rows.

Exact definitions and when each breaks Table

Metric	Definition	Undefined or misleading when
MAE	mean of \|e\|	Ignores volume; its best constant is the median
RMSE	square root of the mean of e squared	A few large misses dominate; its best constant is the mean
MAPE	100 x mean of \|e\| / \|y\|	Any y = 0; explodes near zero; over-forecasts cost more than under-forecasts
sMAPE (0-200 form)	200 x mean of \|e\| / (\|y\| + \|forecast\|)	A row where actual and forecast are both zero; still asymmetric
WAPE	100 x sum of \|e\| / sum of \|y\|	All actuals zero
MASE	MAE / in-sample MAE of a naive forecast at lag m	Training series constant at lag m
Pinball loss at q	mean of max(q e, (q - 1) e)	Meaningful only for a forecast meant as the q-quantile
R-squared	1 - sum of e squared / sum of (y - mean of y) squared	Constant test target; moves with test variance; negative when worse than the mean

e = y - forecast on evaluation rows after the cutoff (Day 15). Definitions follow Hyndman and Koehler (2006) and FPP3 section 5.8; WAPE is a practitioner measure. sMAPE was meant to fix MAPE's asymmetry and does not; FPP3 recommends against it.

Derivation: which statistic each loss minimizesadvanced8 minSquared error to the mean, absolute error to the median, pinball to the quantile, MAPE to a low weighted median, and unit prices to a quantile.

Derive what each loss minimizes, in one line each Formula

\frac{d}{dc}\,\mathbb{E}(y-c)^2 = -2\,\mathbb{E}(y-c), \quad \frac{d}{dc}\,\mathbb{E}\lvert y-c \rvert = P(y<c) - P(y>c), \quad \frac{d}{dc}\,\mathbb{E}\,L_q = P(y<c) - q

$\frac{d}{dc}$: set each derivative to zero to find the constant forecast c with the lowest expected loss
$\mathbb{E}(y-c) = 0$: squared error: solved by c = the mean
$P(y<c) = P(y>c)$: absolute error: as much probability above as below, so c = the median
$P(y<c) = q$: pinball loss at level q: c = the q-quantile

A loss is a contract about which summary of the outcome the forecast should be. The table below pushes five numbers through all three contracts by hand.

Derivation: set the slope of the expected loss to zero Sequence

The risk of a constant
R(c) = E L(Y, c): the expected loss of forecasting the same c for every row. On data, the average over rows.
Squared error
E(Y - c)^2 = Var(Y) + (EY - c)^2. Only the second term depends on c, so c = EY, the mean; the floor left over is the variance.
Absolute error: the slope
Raise c by a small h: rows below c each cost h more, rows above each cost h less. Slope = P(Y < c) - P(Y > c).
Absolute error: the minimum
The slope is zero with half the probability on each side: c = the median. With an even n, any c between the middle two values ties.
Pinball at level q
Rows below c cost (1 - q) per unit, rows above cost q. Slope = (1 - q) P(Y < c) - q P(Y > c) = P(Y < c) - q.
Pinball: the minimum
Zero at P(Y < c) = q: the q-quantile. At q = 0.5 the pinball loss is half the absolute error, so the median again.
MAPE
The absolute-error argument with row weights 1 / y: the slope is zero at the weighted median, pulled toward the small actuals.
Prices to a quantile
c_u (y - c)+ + c_o (c - y)+ is (c_u + c_o) times pinball at q = c_u / (c_u + c_o): the cheapest forecast is that quantile.

Standard results for a continuous outcome; with ties or few rows the minimizer can be an interval. Each step is checked numerically on five numbers in the next chapter.

MAPE is an absolute error with weights 1 / actual Formula

\mathrm{MAPE}(c) = \frac{100}{n}\sum_{t} \frac{1}{y_t}\,\lvert y_t - c \rvert

$\lvert y_t - c \rvert$: the absolute error of row t
$1 / y_t$: a weight that is large for small actuals: a 2-unit store-day counts 50 times a 100-unit one
$\arg\min_c$: the weighted median with weights 1 / y_t, which sits below the ordinary median

Same algebra as MAE with unequal weights; that is why MAPE-selected forecasts drift low on right-skewed demand.

Pinball (quantile) loss Formula

L_q(y, \hat y) = \max\big(q\,(y - \hat y),\; (q - 1)\,(y - \hat y)\big)

$q$: the quantile level the forecast is meant to be
$y > \hat y$: under-forecast: costs q per unit
$y < \hat y$: over-forecast: costs 1 - q per unit

Its expected value is minimized by the q-quantile of the outcome (Gneiting, 2011); q = 0.5 gives half the absolute error.

Asymmetric cost is a scaled pinball loss Formula

c_u\,(y - \hat y)_+ + c_o\,(\hat y - y)_+ = (c_u + c_o)\, L_q(y, \hat y), \quad q = \frac{c_u}{c_u + c_o}

$c_u$: cost of one unit under-forecast, e.g. EUR 12 of margin lost on a stock-out
$c_o$: cost of one unit over-forecast, e.g. EUR 8 of markdown on an unsold unit
$q$: the cost-minimizing quantile (critical ratio): 12 / 20 = 0.6 here

The unit costs are assumptions to replace with finance's numbers. Energy: a day-ahead shortfall priced at three times a surplus gives q = 3 / 4 = 0.75 (hypothetical prices). The same identity gives Day 13's cost-based threshold; Day 26 returns to quantile forecasts.

The pinball loss finds the quantile, and the price ratio picks it Illustration

Each curve bottoms out where a share q of the actuals lies below the forecast: the empirical q-quantile (ticks on the axis).

Cost is lowest at 39.8 units; the empirical 0.8-quantile is 39.9. Forecasting the median costs EUR 536 more per period, the mean EUR 283 more.

Synthetic data: 400 seeded log-normal demands (median 26.2 units). Unit costs are assumptions: a stock-out priced at four times an unsold unit. Losses computed with the app's pinball-loss function.

Left: each pinball curve bottoms out at its own empirical quantile. Right: a stock-out priced at four times an unsold unit moves the cheapest forecast to the 0.8-quantile, above mean and median. Score a quantile forecast with pinball at its own q and check coverage: about a share q of actuals should fall at or below it.

Worked by hand: five numbers through three losses6 minPush five illustrative numbers through MAE, MSE, pinball and MAPE on paper, find each best constant, and see the same thing as a sweep.

Five numbers, three losses, three best forecasts (by hand) Table

Constant c	MAE	MSE	Pinball at q = 0.75
9 (the median)	7.8 (lowest)	156.6	5.15
12	8.4	135.6	4.70
14 (mean and 0.75-quantile)	8.8	131.6 (lowest)	4.40 (lowest)
36 (the maximum)	22.0	615.6	5.50

Daily return parcels at an online retailer: 4, 7, 9, 14, 36 (illustrative). MAE at c = 9 by hand: absolute errors 5, 2, 0, 5, 27 sum to 39, over 5 rows = 7.8. Each column bottoms out at its own statistic; verified in Python.

The five-number example, worked line by line Sequence

Sort and summarize
Returns per day: 4, 7, 9, 14, 36. The middle value (median) is 9; the mean is 70 / 5 = 14.
MAE at the median
Absolute errors 5, 2, 0, 5, 27 sum to 39, so MAE = 7.8. Moving c off 9 adds error on three rows and saves it on two.
MSE at the mean
Errors -10, -7, -5, 0, 22; squares 100, 49, 25, 0, 484 sum to 658, so MSE = 131.6, exactly the variance of the five numbers.
MSE elsewhere
MSE(c) = 131.6 + (14 - c)^2: c = 12 gives 135.6 and c = 9 gives 156.6, as in the table.
Pinball q = 0.75 at 14
One row above: 0.75 x 22 = 16.5. Three rows below: 0.25 x (10 + 7 + 5) = 5.5. Total 22 over 5 rows = 4.4.
Why 14 is the 0.75-quantile
3 of 5 rows (0.6) lie below 14 and 4 of 5 (0.8) at or below it, so P(Y < c) crosses 0.75 exactly at c = 14.
MAPE's best constant
Weights 1 / y, normalized: 0.41, 0.24, 0.18, 0.12, 0.05. The running total passes 0.5 at y = 7, so the MAPE-best constant is 7.
Check
MAPE is 45.6% at c = 7 and 52.9% at c = 9: below the median, as derived. All values recomputed with numpy on a fine grid of c.

Illustrative numbers for an online retailer's returns desk. On skewed demand the three best constants (9, 14, 7) differ by whole units, not decimals. Robustness: drop the 36 and the mean falls from 14 to 8.5 while the median moves only from 9 to 8. Redo it on paper, then compare with the sweep picture below.

One sample, three metrics, three different best forecasts Illustration

Median 8.0, mean 10.3: a long right tail pulls the mean up.

Straight pieces with a kink at every actual: lowest at the median, where as many rows sit above as below.

Smooth bowl: lowest at the mean, so the tail drags the best forecast up.

Lowest below the median: each row weighs 1 / actual, so small actuals pull the best forecast down.

Sweep a single constant forecast over skewed demand. MAE bottoms out at the median, RMSE at the mean, MAPE below both. Choosing the metric chooses the number you will forecast.

Properties: consistency, propriety, and where each loss breaks7 minWhat the average loss converges to, why a proper score rewards honesty, MSE as variance plus bias squared, and the diagnostic for each failure.

Why a proper score cannot be beaten by lying Formula

\mathbb{E}\big[(p - Y)^2\big] = \pi\,(1-p)^2 + (1-\pi)\,p^2, \qquad \frac{d}{dp} = 0 \;\Rightarrow\; p = \pi

$\pi$: the true probability of the event Y = 1
$p$: the probability the model reports
$(p-Y)^2$: the Brier score of one outcome; its expectation is minimized only at p = pi (strictly proper)

Derivative -2 pi (1 - p) + 2 (1 - pi) p = 0 gives p = pi. Log loss is strictly proper too; accuracy and the absolute error of a probability are not, so both reward shading the report.

Honesty pays under Brier, rounding pays under absolute error Table

Reported p	Expected Brier score	Expected absolute error
0.0 (claim impossible)	0.300	0.300 (lowest)
0.3 (the truth)	0.210 (lowest)	0.420
0.5 (hedge)	0.250	0.500

Computed from the formula above with true probability 0.3. The absolute error of a probability is minimized by rounding to 0 or 1, so it invites overconfident reports; score probabilities with Brier or log loss plus a reliability table.

Where each score breaks, and the check that catches it Table

Score	Breaks when	Check
Squared error	A few spikes carry the total; its standard error sd(row loss) / sqrt(n) stays wide	Share of total squared error in the top 1% of rows
MSE = variance + bias squared	A shrunk forecast trades a little bias for less variance	Report the mean error beside it
Absolute error of a probability	Rounding to 0 or 1 wins: a readmission model stops saying 0.3	Brier or log loss, plus a reliability table
MAPE	Actuals near zero; forecasts shaded low	Count actuals near zero; report the mean error
Pinball at q	The forecast was meant as a different quantile	Empirical coverage against q
Any score	Not consistent for the statistic asked for: MAE can prefer a worse mean forecast	Name the statistic each model targets

With independent rows the average loss converges to the risk (consistency). A score ranks forecasts honestly only if it is consistent for the statistic you asked for (Gneiting, 2011).

Scale an unbiased forecast down and MAPE improves Illustration

MAPE is lowest with every forecast cut to 80% of the true mean, about 14 units short per row. WAPE, which targets the median, stops at k = 0.92; RMSE, which targets the mean, at k = 0.96.

Synthetic data: 300 rows (store-days), 100 each with true mean 8, 40 and 160 units; actuals are Poisson draws around the mean times a log-normal factor (sd 0.3, mean 1). Metrics computed with the app's metric functions.

Every forecast is the true mean of its row (a store-day, a ward-day) times k. Only MAPE keeps rewarding a cut well below k = 1; this is how a MAPE target turns into a low-biased forecast.

Four ways percentage errors mislead Overview

Zero actuals
One y = 0 makes MAPE undefined. Report "undefined"; never 0, never a silently dropped row.
Asymmetry
With non-negative forecasts an under-forecast costs at most 100%; an over-forecast has no cap.
Equal weights
An 8-unit corner store and a 160-unit flagship count the same in MAPE.
Intermittent demand
Spare parts sell zero most days: use WAPE, MASE or cost, and report bias directly.

Scaled measures, zeros and library traps4 minWAPE and MASE written out, when each is undefined, and a library default that hides zeros.

WAPE is MAPE with volume weights Formula

\mathrm{WAPE} = \frac{\sum_t \lvert e_t \rvert}{\sum_t \lvert y_t \rvert} = \sum_t w_t \frac{\lvert e_t \rvert}{\lvert y_t \rvert}, \quad w_t = \frac{\lvert y_t \rvert}{\sum_s \lvert y_s \rvert}

$w_t$: row t's share of total volume; MAPE uses the equal weight 1/n instead
$\frac{\lvert e_t \rvert}{\lvert y_t \rvert}$: absolute percentage error of row t

Multiply by 100 for a percentage. Compute it as the ratio of sums - a zero actual then only adds its error to the numerator and no longer breaks the average.

MASE, scaled by a naive forecast Formula

\mathrm{MASE} = \frac{\frac{1}{n}\sum_{t} \lvert e_t \rvert}{\frac{1}{T-m}\sum_{t=m+1}^{T} \lvert y_t - y_{t-m} \rvert}

$\frac{1}{n}\sum_t \lvert e_t \rvert$: MAE of the model on the evaluation rows
$y_t - y_{t-m}$: error of the seasonal naive forecast on the training series (m = 1 without seasonality)
$\mathrm{MASE} < 1$: the model beats that naive method's in-sample error on average

Scale-free, so it compares models across series of different size, and defined with zero actuals in the evaluation window; undefined only if the training series is constant at lag m.

Training objective, evaluation metric and library defaultsadvanced3 minFit with the loss that matches the decision; the defaults fit the mean, and one quantile default is not the median.

Fit the statistic the metric rewards, or the model aims at the wrong number Diagram

Decision's statisticmean, median or a quantile q
chooses
Training objectivedefaults fit the mean
Metric rewards it too?
- yes: to Score and report
no
Retrain or post-shifte.g. an L2 fit judged by pinball at 0.8
score again
Score and reportwith bias and cost beside it

Before comparing two models, ask which statistic each was trained for and which one the metric rewards. With mostly zero demand, report bias and cost beside any absolute-error score.

Train on the loss you will be judged by (installed versions) Table

Target of the forecast	scikit-learn 1.9 HistGradientBoosting	LightGBM 4.7	Score with
Mean	loss='squared_error' (the default)	objective='regression' (the default, L2)	RMSE or MSE
Median	loss='absolute_error'	objective='l1'	MAE; WAPE across series
Quantile q	loss='quantile' with quantile=q (no default)	objective='quantile', alpha=q (default 0.9)	mean_pinball_loss(alpha=q); its default is 0.5
Counts, non-negative	loss='poisson'	objective='poisson'	mean_poisson_deviance, plus WAPE

Checked against the installed scikit-learn 1.9.1 and LightGBM 4.7.0. XGBoost 3.4 offers objective='reg:quantileerror' with quantile_alpha. Always pass q explicitly. Poisson or Tweedie objectives keep count forecasts non-negative; score them with deviance plus a business metric.

Classification and ranking metrics3 minWhich question each classification or ranking metric answers, linked to the cost-based threshold of Day 13.

Classification and ranking metrics (Day 13 recap) Table

Metric	Answers	Read against	Does not tell you
Accuracy	Share of correct labels at a threshold	The majority-class rate	Rare-class performance; unequal costs
Precision and recall	Errors at the chosen threshold	The cost-based threshold (Day 13)	Behaviour at other thresholds
ROC-AUC	How well scores rank positives above negatives	0.5, a random ranking	Precision at your operating point; calibration
PR-AUC (average precision)	Precision across recall levels	The prevalence, a random ranking	Calibration; the operating point
Log loss, Brier score	Quality of the probabilities	Always predicting the base rate	Which threshold to act on
Precision@k, NDCG	Quality of the top k of a ranked list	The list a team can act on	Anything below position k

Set the threshold from costs first (Day 13), then read precision and recall at it. A collections team that can phone 50 customers a day reads precision@50, not ROC-AUC. When the probability itself is the product (a credit limit, a premium, a ward's staffing), score it with log loss or Brier, which are strictly proper, plus a reliability table.

Aggregation: pooled, macro and weighted3 minHow the same errors rank two models in opposite orders depending on how segments are combined.

Same errors, opposite ranking: pooled versus averaged by class Chart

A, pooled (volume-weighted)11.3 % WAPE
B, pooled (volume-weighted)16 % WAPE
A, mean of class WAPEs14.6 % WAPE
B, mean of class WAPEs12.2 % WAPE

Computed from the application data. Pooling lets the large class carry the result (A wins); averaging the three classes equally lets the small class vote (B wins).

Three ways to aggregate a metric Table

Aggregation	How	Dominated by	Use when
Micro (pooled)	Sum errors and volumes over all rows, then divide	Large classes and series	The total is what gets paid for
Macro	Compute per class or series, then average equally	Small classes and series	Each segment matters equally, such as a rare class
Weighted	Average per segment with stated weights	Whatever the weights favour	Weights reflect money or volume; state them

Choose the aggregation from who pays for the errors, and show the segment breakdown next to any single number. scikit-learn's average='micro', 'macro' and 'weighted' (weights = class support) apply the same logic to precision, recall and F1.

Is the difference real? Paired intervalsadvanced4 minPut an interval on a metric difference with a paired bootstrap, keep time structure with blocks, and know the classic test.

A paired bootstrap for a metric difference Sequence

Pair the rows
Score both models on the same rows: two separate intervals can overlap and still hide a clear paired difference.
Resample
Draw rows with replacement; with autocorrelated errors draw blocks of consecutive days, or the interval comes out too narrow.
Recompute
The metric difference on each resample, for example 10,000 times.
Read the interval
The middle 95% of the differences: does it exclude zero and the margin that matters?

The classic alternative is the Diebold-Mariano test: it compares two forecasts on the loss differential and allows asymmetric losses and autocorrelated errors. In a time-ordered backtest, also report the spread across folds (Day 15).

Is B really different? Paired bootstrap on the application data Table

B minus A	Point estimate	95% interval
Euros per departure	+260	+132 to +392
MAPE, percentage points	-2.7	-4.6 to -0.8

Computed: 10,000 paired resamples of the 300 departures. Both differences are clear and point in opposite directions: uncertainty does not pick the metric, the decision does.

Goodhart's law and guardrails2 minWhat happens to a metric once it becomes a target, and how to pair it with a check.

Once a metric is a target, its cheapest moves get found Loop

Metric is a targeta bonus or a release gateinvites ↓
Cheap move foundunder MAPE: shade every forecast downproduces ↓
Metric improvesbut the cost does notcaught by ↓
Guardrail firesone the cheap move breaks: bias for MAPEprompts ↓
Target revisedtoward cost, fixed window and segmentssets a new ↓
↺ back to 1 · Metric is a target

Pair each target with a guardrail that its cheapest moves would break. The table lists typical drifts and guardrails per target.

Targets, how they drift, and a guardrail Table

Target	Typical drift	Guardrail
MAPE below a threshold	Forecasts biased low; awkward series dropped	Mean error (bias) and WAPE on the full population
ROC-AUC	Gains in parts of the ranking nobody acts on	Precision at the operating threshold or at k
Accuracy on a rare event	Predicting the majority class	Recall on the rare class; cost per decision
Cost per decision	Costs counted on a convenient window or segment	A fixed window and population, reported by segment
Claims-handling time per file	Hard claims deferred or routed elsewhere	Backlog age and the share of reopened claims

Practitioner heuristics about incentives, not measured effects.

Memory anchor

Metric, failure condition, and price per error - and an undefined value is never a zero.

Useful for

Choosing a primary metric and its guardrails for a forecast or classifier before modeling starts.
Translating "the metric improved" into units, money and risk for a stakeholder, from store replenishment to claims reserving.
Auditing a model report for undefined values, hidden zeros and metrics that ignore volume.

Not suitable for

Replacing an experiment - a cost metric computed offline still assumes the model's outputs are acted on as modeled.
Comparing models across series with different units or meanings of zero without a scaled or relative measure.

Traps

Averaging percentages across unequal volumes

A mean of per-row or per-series percentage errors lets many small units outvote the few that carry the money, and a single zero actual makes it undefined.

How to avoid it

Use a ratio of sums (WAPE) or error in business units, and report a breakdown by volume class.

Treating the target metric as the goal

Once a metric is a bonus or a release gate, models and teams drift toward whatever improves it cheaply, such as biased low forecasts under MAPE.

How to avoid it

Pair every target with a guardrail that exposes gaming and revisit the metric against cost each quarter.

Senior reasoning

A senior starts from the decision and its cost, then picks the cheapest metric that preserves the ranking in money. They know each metric's failure condition, report undefined values as undefined, break any headline number into volume segments and direction of bias, and treat a metric that has become a target as suspect until it is checked against cost.

Interview question

"Your forecast model's MAPE dropped from 15% to 12%. Is it better?" How would you structure the answer?

Outline your answer first.

Recall cards scheduled after completion

When is each of MAPE, sMAPE, WAPE, MASE and R-squared undefined, and what do you report?
How do you turn a metric change into a business number, and what is Goodhart's law?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Another look at measures of forecast accuracy — Rob J. Hyndman, Anne B. Koehler, 2006core
Defines MASE as MAE scaled by the in-sample naive forecast error and critiques percentage and relative measures; basis for the MASE row and the zero conditions.
Forecasting: Principles and Practice (3rd edition) — Rob J Hyndman, George Athanasopoulos, 2021core
Section 5.8 gives the exact MAE, RMSE, MAPE, sMAPE and scaled-error definitions, the zero and asymmetry caveats for percentage errors, and the recommendation against sMAPE. (5.8 Evaluating point forecast accuracy)
Quantiles as optimal point forecasts — Tilmann Gneiting, 2011optional
Pinball (quantile) loss is the scoring rule whose minimizer is the quantile, which links asymmetric costs to a forecast quantile.
Making and Evaluating Point Forecasts — Tilmann Gneiting, 2011optional
A scoring function must be consistent for the functional being forecast (mean, median, quantile); otherwise average scores can rank forecasts wrongly. Basis for the sweep and objective chapters.
Comparing Predictive Accuracy — Francis X. Diebold, Roberto S. Mariano, 1995optional
A test of equal accuracy for two forecasts that allows non-quadratic, asymmetric losses and serially correlated errors.
Categorizing Variants of Goodhart's Law — David Manheim, Scott Garrabrant, 2018optional
Categorizes how optimizing a measure can separate it from the goal; supports the Goodhart discussion (preprint, not peer reviewed).
Metrics and scoring: quantifying the quality of predictions (scikit-learn User Guide 3.4) — scikit-learn developersoptional
Defines accuracy, precision, recall, F1, ROC-AUC, average precision, log loss and Brier score in one place for the classification summary. (3.4 Metrics and scoring)

Optional sections

Deep dive · 10 min

Why MAPE favours low forecasts. For an actual $y>0$ an over-forecast can produce a percentage error above 100%, but an under-forecast is capped at 100% for non-negative forecasts (a forecast of zero). Across many rows the MAPE-minimizing constant is therefore pulled below the median, and a model tuned on MAPE drifts low. The effect is documented as a heavier penalty on negative errors in FPP3 section 5.8; check bias directly (mean error) before adopting a MAPE-selected model.

From cost to pinball loss. With a cost of $c_u$ per unit under-forecast and $c_o$ per unit over-forecast, cost per row is $c_u\,(y-\hat y)_+ + c_o\,(\hat y-y)_+$ . Dividing by $c_u+c_o$ gives pinball loss at $q=c_u/(c_u+c_o)$ , so the cost-minimizing point forecast is the $q$ -quantile of demand, not the mean (Gneiting, 2011; Day 26 returns to this). If a lost sale costs EUR 12 of margin and a leftover unit EUR 8 of markdown, $q=0.6$ .

Classification in one paragraph. Pick the threshold from costs (Day 13), then report precision and recall at that threshold, PR-AUC against prevalence, and log loss and calibration for the probabilities. ROC-AUC says how well the scores rank positives above negatives and is insensitive to prevalence; it says nothing about precision at your operating point. Calibration is checked on held-out data with a reliability table: bin scores, compare the mean score with the observed rate per bin.

Junior / mid / senior ladder for "MAPE decreased". Junior: better, ship. Mid: which rows moved, zeros, bias, WAPE and MAE agree? Senior: what decision consumes the forecast, what is the cost of a unit in each direction, does the gain hold by segment and period, and what will the team do next to push the number? The ladder rises from reading the number to reading the incentives around it.

Hands-on lab: The full metric suite on one series, with zeros and a test-window change · 15 min

Run the script. It builds a synthetic daily series with weekly seasonality, one closure day with zero bookings in the test period, and two forecasters (weekday means from training, and the same forecast inflated by 15%). It prints MAE, RMSE, MAPE, sMAPE, WAPE, MASE (m = 7), R-squared and pinball loss at q = 0.8 for each. Then answer: (a) which metrics are undefined and why; (b) does the biased forecast rank last under every metric; (c) how much does R-squared of the same model change between the 28-day window and its first 14 days? The seasonal-naive line shows MASE below 1 against the in-sample naive scale.

import numpy as np

rng = np.random.default_rng(160)
t = np.arange(56 + 28)                                    # 8 training weeks, 4 test weeks (synthetic route)
season = np.array([1.0, 0.9, 0.9, 1.0, 1.2, 1.4, 1.3])[t % 7]
y = rng.poisson(40 * season).astype(float)
y[70] = 0                                                 # one closure day in the test period
train, test = y[:56], y[56:]

def undefined(x): return "undefined" if x is None else f"{x:.2f}"

def suite(y, f, train, m=7, q=0.8):
    e = y - f
    mape = None if (y == 0).any() else 100 * np.mean(np.abs(e) / np.abs(y))
    den = np.abs(y) + np.abs(f)
    smape = None if (den == 0).any() else 100 * np.mean(2 * np.abs(e) / den)
    wape = None if np.abs(y).sum() == 0 else 100 * np.abs(e).sum() / np.abs(y).sum()
    scale = np.mean(np.abs(train[m:] - train[:-m]))           # in-sample seasonal-naive MAE
    mase = None if scale == 0 else np.mean(np.abs(e)) / scale
    sst = ((y - y.mean()) ** 2).sum()
    r2 = None if sst == 0 else 1 - (e ** 2).sum() / sst
    pin = np.mean(np.where(e >= 0, q * e, (q - 1) * e))      # pinball loss at level q, f read as the q-quantile
    return {"MAE": np.abs(e).mean(), "RMSE": np.sqrt((e ** 2).mean()), "MAPE": mape, "sMAPE": smape,
            "WAPE": wape, "MASE": mase, "R2": r2, "pinball0.8": pin}

f_naive = np.concatenate([train[-7:]] * 4)                         # repeat the last training week
dow = np.array([train[np.arange(56) % 7 == d].mean() for d in range(7)])
f_mean = dow[np.arange(28) % 7]                                     # weekday averages from training only
for name, f in [("seasonal naive", f_naive), ("weekday mean", f_mean), ("weekday mean x1.15", 1.15 * f_mean)]:
    print(f"{name:<20}", {k: undefined(v) for k, v in suite(test, f, train).items()})
print("zero days in test:", int((test == 0).sum()), "| test variance:", round(test.var(), 1))
calm = test[:14]                                                    # same model, a calmer evaluation window
print("R2 weekday mean, 28 days:", undefined(suite(test, f_mean, train)["R2"]),
      "| first 14 days:", undefined(suite(calm, f_mean[:14], train)["R2"]))

Interview practice · 10 min

Finance at a grocery chain wants demand-forecast accuracy reported as a single percentage across 400 stores of very different size. What do you propose?

Answer as you would in the interview.

Content version 7 · last verified 2026-10-11 · depends on numpy