Skip to content
ML Academy

Core day 26 of 60 · Week 6 · Part I · evaluation depth

Advanced Forecasting

Revenue management protects seats for a higher fare class using a pickup forecast with an "80% interval". Last quarter only 43% of outcomes fell inside it. What does that promise mean, what broke it, and how should the planning number change when an empty protected seat and a diluted seat cost different amounts?

By the end: From a set of forecast errors you build a band, check its coverage and width using only errors that existed at the cutoff, choose the planning quantile from a cost ratio, and say which assumption a stale calibration violates.

At a glance

A forecast band is a promise about coverage; choose the planning quantile from the costs and recheck the promise wherever the regime can move.

  • Three objects

    Confidence interval for a mean, prediction interval for one outcome, quantile for a decision.

  • Coverage and sharpness

    Calibrated first, then as narrow as possible; read the two together.

  • Quantile from costs

    tau* = c_under / (c_under + c_over) sets stock, staff or a reserve; score it with pinball loss.

  • Conformal

    Past errors set the width; the guarantee needs exchangeable errors.

  • Monitor coverage

    A shift drops coverage first; recalibrate on errors known at each cutoff.

Recall · 5 min

Without notes: (1) at a D-14 cutoff for departure date T, which earlier departures have a known forecast error? (2) Which point forecast minimizes MAE, and which loss has the 0.9 quantile as its minimizer? Then clear your due review cards.

Open due reviews

Sign in to record this block.

Concept · 10 min

A point forecast of "100 more bookings" is one number; the decision needs the spread and the side that hurts more. Judge a band on two things that pull apart: coverage (an "80%" band that caught 43% is miscalibrated) and sharpness (an enormous band always covers). Aim for the sharpest band that is still calibrated.

A band that looks sharp can cover far less than it promises, and recalibrating on recent errors recovers part of it

Predict first

A band is rebuilt every week from the last 100 errors. A competitor adds capacity today. What happens to coverage over the next two weeks?

Text description of the figure

A simulated series with controls for confidence level, noise regime (constant, heteroscedastic, regime change) and horizon, and a selector for the method (Gaussian, empirical residuals, split conformal). The band and the actuals inside and outside it are drawn; a table reports coverage and mean width for all three methods, and a rolling coverage chart shows it falling away from the nominal level.

Three objects that get confused Comparison
CriterionConfidence intervalPrediction intervalQuantile forecast
CoversAn unknown mean or parameter (Day 4)One future outcome, including its own noiseThe value outcomes stay below with probability tau
Narrows with more data?Yes, towards zero widthOnly down to the noise levelConverges to the true quantile
ExampleRetail: the average Friday basket value across all storesEnergy: tomorrow's peak load at one substationInsurance: the 0.9 quantile of next quarter's claims, to reserve

An 80% prediction interval is the pair of quantiles at 0.1 and 0.9. The airline code below plans at the 0.27 quantile of pickup.

Coverage and sharpness pull apart Map

Honest but vague

Calibrated and wide: safe, costly to plan with.

The goal

Calibrated and as narrow as the data allow.

Off-centre

Wide and still missing: the forecast itself is biased.

Confidently wrong

Narrow and miscalibrated: the most dangerous band.

Maximize sharpness subject to calibration (Gneiting, Balabdaoui and Raftery, 2007).

Choose the planning quantile from the costs Formula
τ∗=cucu+co=3030+80≈0.27\tau^{*} = \frac{c_u}{c_u + c_o} = \frac{30}{30 + 80} \approx 0.27
cuc_u
cost per seat of planning too low: dilution, EUR 30 (hypothetical)
coc_o
cost per seat of planning too high: spoilage, EUR 80 (hypothetical)
τ∗\tau^{*}
the quantile that minimizes expected cost; 0.5 only when the costs are equal

The same rule sets a nurse roster (understaffing dearer than overstaffing) or an energy trader's bid; only c_u and c_o change. The critical ratio is only as reliable as those two unit costs.

The cheapest plan is a quantile, and the saving grows with the asymmetry Illustration
6080100120140Outcomes and two plansmedianplan 93Pickup, seats

Simulated pickup outcomes. Planning at the median leaves as much room above as below; with spoilage the dearer error, the cheapest plan sits low, at the 0.27 quantile.

30050070000.250.50.751Expected cost by plantau* = 0.27EUR 443median: EUR 527Planning quantile tauEUR per departure

Expected cost per departure for every planning quantile. The minimum lands at 0.28 (theory: 0.27); planning at the median costs EUR 84 more per departure here.

0.000.250.500.751.000.1110Best quantile and saving vs cost ratio30 vs 80best quantile tau*cost saved vs medianr = c_under / c_over (log scale)

Change the cost ratio and the best quantile moves along tau* = r / (1 + r); the saving over planning at the median grows quickly once one error costs several times the other.

Synthetic: 4,000 pickup outcomes, mean 100 and sd 12 seats (seeded). Hypothetical costs EUR 30 (too low) and EUR 80 (too high) per seat; the saving curve is relative to planning at the median.

Expected cost computed for every planning quantile on simulated outcomes; the minimum sits where the cost ratio says.

What a jump in noise does to an 80% band Illustration
150200250Rolling Gaussian 80% band, outcomes and missesnoise sd x2.6 from hereTime step (one-step-ahead forecasts)
  • Gaussian band
  • adaptive conformal band (dashed)
  • outcome outside the Gaussian band

Before the break the band (shaded) catches about 8 in 10 outcomes. After it, the noise is 2.6 times larger and the band is still sized from old errors: misses (red) pile up until the window refills.

40%60%80%100%150200250Coverage over the last 30 stepssplit conf.GaussianadaptivebreakTime step

Trailing 30-step coverage; dashed line: the nominal 80%. All three sag after the break; the adaptive band reacts within a few misses because every miss lowers its target miscoverage.

After the breakallfirst 40mean widthGaussian (rolling)before: 79%, width 3.068%53%6.4Split conformal (rolling)before: 79%, width 3.066%48%6.1Adaptive conformalbefore: 80%, width 3.382%73%9.1

After the break: coverage over all later steps and over the first 40, with mean width. Coverage is bought with width; the adaptive band pays more of it, sooner.

Synthetic local-level series (seed 26), noise sd 1 then 2.6 from step 186; SES one-step forecasts; 80% bands from the last 100 errors (interval-lab engine). Adaptive conformal uses gamma = 0.05.

Rolling bands keep the old width until the window refills; adaptive conformal reacts to its own misses, recovers coverage sooner and pays with a wider band.

Cost per departure, EUR (today's run, hypothetical unit costs) Chart
  • Before: plan at the mean497 EUR
  • Before: plan at q* (static)the asymmetry pays440 EUR
  • After: plan at the mean647 EUR
  • After: q* from a stale bandGaussian: 836804 EUR
  • After: q* from the rolling band629 EUR

Leaning towards one side amplifies a level error, so a stale quantile plan is worse than ignoring the asymmetry.

  • Three objects: a confidence interval (a mean), a prediction interval (one future departure), a quantile forecast (the value outcomes stay below with probability tau).
  • Quantile from costs: the newsvendor critical ratio, scored with pinball loss.
  • Band from past errors: conformal prediction gives marginal coverage under Exchangeability; shifts break it.
  • More pieces: reconciliation, combinations, intermittent demand.

Assumptions

  • Calibration errors are exchangeable with future errors. Seasonality, drift and breaks violate this, and the guarantee is marginal, averaged over departures, not per route or month.

  • Calibration errors existed at the cutoff (at D-14, only departures that have flown) and match the planning horizon.

  • The unit costs are known; the critical ratio is only as reliable as c_under and c_over (hypothetical here).

Sign in to record this block.

Application and workplace decision · 10 min

Run the simulation (synthetic). Predict which band keeps 80% coverage after the shift.

  • Pickup follows a weekday pattern; the forecast is the weekday mean of departures 0-199.
  • Departures 200-299 calibrate the bands; 300-599 are the test, with pickup up 18% from departure 450 when a competitor leaves.
  • Forecasts are issued at D-14, so rolling calibration may use only departures that have flown.
  • Three 80% bands (Gaussian, static empirical, rolling empirical), and the cost of planning at the mean versus the cost quantile (hypothetical EUR 30 dilution, EUR 80 spoilage per seat).

Python

import numpy as np
from scipy.stats import norm

rng = np.random.default_rng(26)
n, fit, test, shift, lag = 600, 200, 300, 450, 14   # departures in date order; forecasts issued at D-14
mu = 100 + 12 * np.sin(2 * np.pi * np.arange(n) / 7)  # weekday pickup level, seats (synthetic)
mu[shift:] *= 1.18                                  # a competitor exits at departure 450: pickup +18%
y = rng.normal(mu, 0.12 * mu)                       # realised pickup per departure
f = np.array([y[:fit][np.arange(fit) % 7 == d % 7].mean() for d in range(n)])  # weekday means, fit period
e = y - f                                           # signed forecast error, seats
c_under, c_over = 30, 80                            # hypothetical EUR per seat: dilution vs spoilage
q_star = c_under / (c_under + c_over)               # newsvendor critical ratio = 0.27

def quantiles(i, method):                           # error quantiles usable at departure i's cutoff
    ps = [0.1, 0.9, q_star]
    if method == "gaussian":
        return norm.ppf(ps) * e[:fit].std()
    pool = e[fit:test] if method == "static" else e[max(fit, i - lag - 99):i - lag + 1]  # flown only
    return np.quantile(pool, ps)

print("method    period  coverage80  width  cost@mean  cost@q*  (EUR per departure)")
for m in ["gaussian", "static", "rolling"]:
    for name, idx in [("before", np.arange(test, shift)), ("after", np.arange(shift, n))]:
        q = np.array([quantiles(i, m) for i in idx])
        inside = (y[idx] >= f[idx] + q[:, 0]) & (y[idx] <= f[idx] + q[:, 1])
        cost = lambda plan: np.where(y[idx] > plan, c_under * (y[idx] - plan), c_over * (plan - y[idx])).mean()
        print(f"{m:9s} {name:6s}  {inside.mean():.2f}  {np.mean(q[:, 1] - q[:, 0]):6.1f}  {cost(f[idx]):8.0f}  {cost(f[idx] + q[:, 2]):7.0f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Sign in to record this block.

Mini-assessment · 8 min

Your first answer is your evidence. You can re-answer later in practice mode; that is labelled practice.

Question 1 · easy

A calibrated 80% prediction interval for tomorrow's pickup on one departure is 84 to 116 seats. What does that mean?
Options

How confident are you? (optional, used for calibration)

Question 2 · medium

A caterer loads meals for a flight. A meal that is loaded and not eaten costs EUR 6 (over-planning). A missing meal costs EUR 14 in vouchers and goodwill (under-planning). The demand forecast for the flight is approximately normal with mean 120 and standard deviation 10 (z for the 0.7 quantile is 0.524). How many meals should be loaded?
meals

How confident are you? (optional, used for calibration)

Question 3 · hard

A team calibrates an 80% split-conformal band on the last 90 days of a stable period. Over the next 60 departures after a competitor adds capacity, only 52% of outcomes fall inside it. A colleague says conformal prediction needs no assumptions, so the model must have degraded. What is the best reading and first action?
Options

How confident are you? (optional, used for calibration)

Question 4 · medium

(Illustrative numbers.) Ten days of demand for a lounge-access add-on on one route: 0, 0, 0, 2, 0, 0, 1, 0, 0, 3 units. Two forecasts are compared by MAE: always 0 (MAE 0.60) and always 0.6, the mean (MAE 0.84). A teammate says the zero forecast is better and should drive stock planning. Which response is right?
Options

How confident are you? (optional, used for calibration)

Question 5 · hard

(Illustrative numbers.) Next quarter's route forecasts sum to 9,400 bookings and the network-level model says 10,100. Planning uses the routes for protection levels and the network figure for capacity. Three routes with a new competitor have lost share of the network this year. Which approach is best?
Options

How confident are you? (optional, used for calibration)

Sign in to complete lessons and keep your progress.

Technical depth · optional · 34 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The quantile, derived from the costs6 minThe pinball loss defined, its minimizer derived step by step, a bakery example worked by hand, how much data a tail quantile needs, and the library defaults that bite.
Pinball (quantile) loss Formula
Lτ(y,q)={τ (y−q)y≥q(1−τ) (q−y)y<qL_{\tau}(y, q) = \begin{cases} \tau\,(y - q) & y \ge q \\ (1-\tau)\,(q - y) & y < q \end{cases}
qτq_{\tau}
the tau-quantile: the smallest q with F(q) >= tau; an 80% prediction interval is the pair q_0.1, q_0.9
qq
the quantile forecast (the plan)
τ\tau
the target quantile; the loss is minimized in expectation by the true tau-quantile
τ=0.5\tau = 0.5
half the absolute error, so the median minimizes MAE (Gneiting, 2011)

At tau* the money cost of a plan equals (c_u + c_o) times its pinball loss.

Why the cost ratio picks the quantile, in four steps Sequence
  1. Nudge the plan up

    Raising q by a small dq adds (1 - tau) dq on the share F(q) of outcomes below q, saves tau dq on the 1 - F(q) above.

  2. The slope

    Net change per unit: dE[L]/dq = (1 - tau) F(q) - tau (1 - F(q)) = F(q) - tau.

  3. Set it to zero

    The expected loss is flat exactly where F(q*) = tau: the minimizer is the tau-quantile, whatever the distribution's shape.

  4. Costs choose tau

    With unit costs, expected money cost = (c_u + c_o) x pinball loss at tau = c_u / (c_u + c_o): the score and the decision agree.

The same argument with c_u and c_o in place of tau and 1 - tau is the newsvendor result; the worked table below pushes real numbers through it.

Worked by hand: a bakery stocks a pastry (hypothetical costs) Table
Plan (units)Expected under-costExpected over-costTotal per day
92 x 6/5 = 2.401 x 1/5 = 0.202.60
102 x 3/5 = 1.201 x 3/5 = 0.601.80
112 x 1/5 = 0.401 x 6/5 = 1.201.60 (minimum)
1201 x 10/5 = 2.002.00

Demand 8, 9, 10, 11 or 12 units, equally likely; c_u = EUR 2 lost margin per unit short, c_o = EUR 1 markdown per unit over. tau* = 2/(2+1) = 2/3, and the 2/3 quantile of demand is 11: exactly where the hand-computed total is lowest (verified in Python).

A tail quantile from 30 errors rests on one or two of them Chart
  • tau = 0.515 errors above it0.22 error SDs
  • tau = 0.86 errors above it0.25 error SDs
  • tau = 0.93 errors above it0.29 error SDs
  • tau = 0.951.5 errors above it0.33 error SDs
  • tau = 0.990.3 errors above it0.43 error SDs

Bar = standard deviation of the sample quantile across 200,000 simulated pools of 30 normal errors (in units of the error SD); note = how many errors sit above it on average. The estimate is consistent, but the further into the tail, the more errors it needs. Computed in Python.

Quantile and interval knobs in the installed libraries (checked 2026-10-12) Table
ToolHow to ask for a quantile or bandDefault that bites
scikit-learn 1.9 QuantileRegressorquantile=tau (default 0.5)alpha=1.0 L1 penalty shrinks coefficients; set alpha=0 for a plain linear quantile
scikit-learn HistGradientBoostingRegressorloss='quantile', quantile=taudefault loss is squared error: no quantile unless asked
scikit-learn GradientBoostingRegressorloss='quantile', alpha=taualpha defaults to 0.9: the name means the quantile here
statsforecast 2.0 ConformalIntervalsprediction_intervals=ConformalIntervals(h, n_windows)n_windows=2, h=1 by default: set h to your horizon

Read from the installed packages (inspect.signature and the source). Separately fitted quantile models can cross; sort or monotonize before use.

Conformal bands and their one assumptionadvanced6 minSplit conformal worked by hand on ten residuals, the rank argument behind its guarantee drawn and simulated, and the adaptive update for shifts.
Split conformal on ten residuals, by hand Table
StepRuleWith the ten residuals
1 CollectAbsolute errors on calibration rows the model never saw1.2, 1.9, 2.4, 3.0, 3.3, 4.1, 4.9, 5.6, 7.0, 9.8 (sorted, seats)
2 Rankk = ceil((n+1)(1-alpha))k = ceil(11 x 0.8) = ceil(8.8) = 9
3 Widthq-hat = the k-th smallest scoreq-hat = 7.0 seats (the 9th value)
4 Bandforecast +/- q-hata forecast of 120 becomes 113 to 127
5 Guaranteecoverage in [1-alpha, 1-alpha + 1/(n+1)]between 80.0% and 89.1%, if the errors are exchangeable

The +1 in the rank is the finite-sample correction: with only 10 errors the 80% band uses the 9th largest score, wider than the plain 0.8 quantile. Rank and bounds verified in Python; the same five steps price a claims reserve or an ETA band as well as a seat forecast.

Why split conformal covers: every one of the n + 1 rank slots is equally likely Illustration
Where can the new |error| land among the ten sorted calibration errors?inside the band: 9 of 11 slots = 81.8%missslot 11/11slot 21/11slot 31/11slot 41/11slot 51/11slot 61/11slot 71/11slot 81/11slot 91/11slot 101/11slot 111/111.21.92.43.03.34.14.95.67.09.8q-hat = 7.0, the k = 9th scoresorted calibration |errors|, seats

Ten scores cut the line into 11 slots. Exchangeable means the new error is just as likely to be the smallest as the largest, so each slot has probability 1/11; the band q-hat = 7.0 covers the first 9: 9/11 = 81.8%, between 80% and 80% + 1/11. The slot widths in seats do not matter, only the ranks.

50%60%70%80%90%100%target 80%Exchangeable, n = 1081.9% (rank theory 81.8%)Exchangeable, n = 10080.2% (rank theory 80.2%)Test errors 1.5x larger, n = 1064.7% (81.8% only if exchangeable)Share of new errors inside the 80% band

20,000 simulated calibration pools per row. The rank argument holds for any error distribution and any n; it fails as soon as the new errors come from a different regime.

Simulation: |standard normal| calibration errors, k = ceil((n + 1) x 0.8); the shifted row draws the new error with 1.5 times the spread, so "rank theory" no longer applies to it. Seeded, 20,000 trials per row.

The ten residuals of the worked example, sorted. Under exchangeability every gap is equally likely; the band covers when the new error lands below the 9th. Simulated check below, with and without a shift. Seasonality, trend and regime shifts break the equal-gap argument (Angelopoulos and Bates, 2023).

The split-conformal width Formula
q^=the k-th smallest of ∣e1∣,…,∣en∣k=⌈(n+1)(1−α)⌉,band: y^±q^\begin{gathered} \hat q = \text{the } k\text{-th smallest of } |e_1|, \dots, |e_n| \\ k = \lceil (n+1)(1-\alpha) \rceil, \qquad \text{band: } \hat y \pm \hat q \end{gathered}
eie_i
forecast errors on n calibration rows
α\alpha
miscoverage: 0.2 for an 80% band

The guarantee is marginal, averaged over departures, not per route or per month. Weighted and adaptive variants exist for shifts (Barber et al., 2023; Gibbs and Candes, 2021).

Adaptive conformal inference, one update per outcome Formula
αt+1=αt+γ (α−errt)errt=1{yt∉C^t(αt)}\begin{gathered} \alpha_{t+1} = \alpha_t + \gamma\,(\alpha - \mathrm{err}_t) \\ \mathrm{err}_t = \mathbf 1\{y_t \notin \hat C_t(\alpha_t)\} \end{gathered}
α\alpha
target miscoverage, 0.2 for an 80% band
αt\alpha_t
the level actually used at step t; a miss lowers it, so the next band widens
γ\gamma
step size: larger reacts faster and makes the width jump after every miss; choose it on a backtest that contains a past shift

Gibbs and Candes (2021) bound the long-run miscoverage gap for any sequence of outcomes; it says nothing about coverage in a given week.

Monitoring coverage as a binomial4 minCoverage is a proportion with a computable standard error; today's before-and-after numbers, the monitoring loop, and a diagnosis path.
Observed coverage is a binomial proportion Formula
p^=XnX∼Binomial(n, 0.8)SE(p^)=0.8×0.2n\begin{gathered} \hat p = \frac{X}{n} \\ X \sim \mathrm{Binomial}(n,\ 0.8) \\ \mathrm{SE}(\hat p) = \sqrt{\frac{0.8 \times 0.2}{n}} \end{gathered}
XX
outcomes inside the band out of n; each departure, order or claim is one draw
n=50n = 50
SE = 0.057: an observed 0.68 sits 2.1 SE below 0.8, a real miss, not noise
n=12n = 12
SE = 0.115: the same 0.68 is within about one SE of 0.8 and proves nothing yet

Judge every coverage number against this band before acting, with draws from the band's own period and segment. Worked values computed in Python; nearby outcomes share shocks, so treat the SE as a lower bound.

Coverage of the 80% bands before and after the shift (today's run) Chart
  • Gaussian, before85 %
  • Gaussian, after43 %
  • Static empirical, before83 %
  • Static empirical, after39 %
  • Rolling empirical, before80 %
  • Rolling empirical, afterrecovers part of the gap69 %

150 test departures in each period; pickup rises 18% from departure 450 (synthetic).

Coverage is a monitored quantity: the loop that keeps a band honest Loop
an 80% band inproduction1Pool errors known atthe cutoffat D-14: flown…2Rebuild the band ona windowlonger window:…3Report coverage andwidthby month, market…430-day coverage in70-90%?outside: alert,…recalibratescorecomparenext cutoff
  1. Pool errors known at the cutoffat D-14: flown departures only, at the planning horizonrecalibrate ↓
  2. Rebuild the band on a windowlonger window: less noise, slower reactionscore ↓
  3. Report coverage and widthby month, market and segment, never one alonecompare ↓
  4. 30-day coverage in 70-90%?outside: alert, then diagnose (next figure)next cutoff ↓
  5. ↺ back to 1 · Pool errors known at the cutoff

Each step repairs one way a band goes stale. The window length is the one dial; it trades reaction speed after a shift against noise in quiet periods.

Coverage is off. Which assumption broke? Diagram
Misses on oneside?Biased pointforecastfix the level first; theband only followsSince a knowndate?Exchangeabilitybrokerecalibrate on recenterrors, or ACIOnly in someslices?Marginal ≠conditionalcalibrate per segment oruse CQRCheck horizon andpoolhorizon errors known atthe cutoffyesnoyesnoyesno
  1. Misses on one side?
    • yes: to Biased point forecast
  2. Since a known date?
    • yes: to Exchangeability broke
  3. Only in some slices?
    • yes: to Marginal ≠ conditional
  4. Check horizon and poolhorizon errors known at the cutoff
  5. Biased point forecastfix the level first; the band only follows
  6. Exchangeability brokerecalibrate on recent errors, or ACI
  7. Marginal ≠ conditionalcalibrate per segment or use CQR

Read coverage with its sampling error first, sqrt(0.8 x 0.2 / n) for an 80% band; at n = 100 that is 0.04.

Reconciliation is one projectionadvanced6 minWhy levels disagree, the projection behind every fix with a three-number example worked by hand, a computed before-and-after, and what each method costs.
Reconciliation, before and after Illustration
180220260300180220260300Base forecasts do not add upcoherent: total = sumSum of route forecastsTotal's own forecast

Each dot is one period: the total's own forecast against the sum of the three route forecasts. Off the diagonal they disagree, by 11.1 on average; every reconciled set lies exactly on it.

RMSE by level, one settingTotal11.312.611.39.6Route A8.58.56.07.1Route B6.96.93.86.4Route C4.14.13.04.0
  • Base (independent)
  • Bottom-up
  • Top-down
  • MinT

Error by level (RMSE) when the total's base forecast has error sd 10 and routes have sd 8, 6 and 4. Bottom-up inherits all route noise in the total; MinT improves every level. Top-down wins on routes only because the simulated shares never move.

0255025102040Total RMSE as the total gets noisierbase totalbottom-upMinTError sd of the total's base forecast (log)RMSE of the total

Turn one knob: how noisy the total's own forecast is. Bottom-up ignores it; the base total gets worse and worse; MinT tracks the better of the two because it weights the levels by their estimated error covariance.

Synthetic hierarchy (total = A + B + C, levels 120/80/50 with a shared season), 60 training and 80 test periods, seed 26. Reconciliation-lab engine: MinT with a shrinkage covariance; top-down uses average historical shares.

Base forecasts disagree across levels; MinT makes them add up and follows whichever level is more accurate.

Every linear reconciliation is one projection Formula
y~=S G y^,GMinT=(S⊤W−1S)−1S⊤W−1\tilde y = S\,G\,\hat y, \qquad G_{\text{MinT}} = (S^{\top} W^{-1} S)^{-1} S^{\top} W^{-1}
SS
summing matrix: rows are total and routes, columns are routes
y^\hat y
base forecasts at every level
WW
covariance of base forecast errors; W = I gives OLS, a diagonal W gives WLS
GG
bottom-up keeps only routes; top-down splits the total by shares
(SG)2=SG(SG)^2 = SG
a projection: any base vector maps to a coherent one, coherent ones pass unchanged; OLS takes the nearest coherent point, MinT measures 'near' with W

Because the result is S times something, every reconciled set adds up. MinT minimizes the trace of the reconciled error covariance (Wickramasuriya et al., 2019).

OLS reconciliation by hand: a chain total and two stores (illustrative) Table
SeriesBase forecastOLS reconciledWhat moved
Chain total10096.67down by gap/3
Store A4043.33up by gap/3
Store B5053.33up by gap/3
Coherencegap = 100 - 90 = +1043.33 + 53.33 = 96.67the 10-unit gap is split equally

G = (S'S)^-1 S' projects the base forecasts onto the coherent subspace: the closest adding-up set in least squares, so each series concedes a third of the gap. A non-identity W (WLS, MinT) leans the split towards the levels with smaller error variance. Verified in Python.

Three ways to make the levels add up, and what each one costs Comparison
CriterionBottom-upTop-downWLS / MinT
UsesRoute forecasts onlyTotal plus historical sharesEvery level, weighted by error variance
The totalSums every route's noiseStable: the total's own forecastLeans on the more accurate level
The routesKeeps full route detailBreaks when shares go staleKeeps detail, pulled towards the total
Unbiased bases stay unbiased?YesNo, once shares moveYes
Hands-on lab RMSE, total / routes55.3 / 32.255.0 / 27.8 (given the true shares)38.5 / 28.8

Hands-on lab, synthetic three-route hierarchy: independent base forecasts score 55.0 / 32.2 and miss adding up by 62.5 on average; every method here closes that gap to 0. Top-down is handed the true shares, its best case. None of them repairs a biased level: check bias per level first.

Four ways to build a band3 minWhat Gaussian, split conformal, CQR and adaptive conformal each assume, and which one your backtest errors call for.
Four ways to build a band, and what each assumes Comparison
CriterionGaussian on residualsSplit conformalConformalized quantile regressionAdaptive conformal (ACI)
AssumesNormal, constant-variance errorsExchangeable calibration and test errorsExchangeability; quantile models for the shapeLong-run coverage only, no exchangeability
Width adapts to the row?No, one width for every rowNo, unless scores are scaledYes: wider where the model expects more spreadNo, it moves one level for all rows
Reacts to a shiftOnly as fast as the residual windowOnly as fast as the calibration windowSame as split conformalYes: every miss lowers the target miscoverage
CostOne standard deviationA held-out calibration blockTwo quantile models plus calibrationOne step size gamma to choose

Plain empirical quantiles of past errors are split conformal without the +1 rank correction. CQR: Romano, Patterson and Candes (2019). ACI: Gibbs and Candes (2021). The guarantee of each is marginal, not per route or month.

Which band? Let the errors decide Hierarchy
stable, normalotherwiseyesnoyesnoBacktest errorsat the horizon: shape,slices, breaksGaussian bandconformal adds little hereSpread varies byslice?segment, season, routeConformal quantilesCQR: wider where spread islargerRegime can move fast?Adaptive conformalACI: long-run coverage,jumpier widthSplit conformalrolling calibration window
  • Backtest errorsat the horizon: shape, slices, breaks
    • stable, normalGaussian bandconformal adds little here
    • otherwiseSpread varies by slice?segment, season, route
      • yesConformal quantilesCQR: wider where spread is larger
      • noRegime can move fast?
        • yesAdaptive conformalACI: long-run coverage, jumpier width
        • noSplit conformalrolling calibration window

Conformal earns its keep when you cannot trust a distribution, not when errors are already normal and stable. The hospital case (quiet weekdays, winter peaks) is the classic reason for a width that adapts to the row.

Forecast combinations, with the variance algebra3 minThe error variance of an average derived, a two-forecast example worked by hand, and the correlation at which averaging stops paying.
Why an average of two forecasts can beat both Formula
Var(e1+e22)=σ12+σ22+2ρσ1σ24w∗=σ22−ρσ1σ2σ12+σ22−2ρσ1σ2\begin{gathered} \mathrm{Var}(\tfrac{e_1 + e_2}{2}) = \frac{\sigma_1^2 + \sigma_2^2 + 2\rho\sigma_1\sigma_2}{4} \\ w^{*} = \frac{\sigma_2^2 - \rho\sigma_1\sigma_2}{\sigma_1^2 + \sigma_2^2 - 2\rho\sigma_1\sigma_2} \end{gathered}
e1,e2e_1, e_2
errors of two unbiased forecasts, e.g. ETS and a GBM for a telecom cell's daily traffic
σ1=10, σ2=12\sigma_1 = 10,\ \sigma_2 = 12
their error SDs (illustrative); rho = 0.5 is the error correlation
equal weights\text{equal weights}
(100 + 144 + 120) / 4 = 91, SD 9.54: below the better member's 10
w∗w^{*}
the variance-minimizing weight on forecast 1: 84 / 124 = 0.68, SD 9.33, barely better than 9.54

The gain comes from rho < 1; at rho = 1 averaging cannot help. Estimated weights carry their own noise, which is why equal weights often hold up on short samples (Bates and Granger, 1969). Numbers checked in Python.

Averaging two forecasts helps only while their errors correlate below 0.65 Chart
-3-2-10100.20.40.60.81break-even rho = 0.65Equal-weight average — Error correlation rho: 0, Average SD minus better SD: -2.19Equal-weight average — Error correlation rho: 0.1, Average SD minus better SD: -1.81Equal-weight average — Error correlation rho: 0.2, Average SD minus better SD: -1.46Equal-weight average — Error correlation rho: 0.3, Average SD minus better SD: -1.11Equal-weight average — Error correlation rho: 0.4, Average SD minus better SD: -0.78Equal-weight average — Error correlation rho: 0.5, Average SD minus better SD: -0.46Equal-weight average — Error correlation rho: 0.6, Average SD minus better SD: -0.15Equal-weight average — Error correlation rho: 0.7, Average SD minus better SD: 0.15Equal-weight average — Error correlation rho: 0.8, Average SD minus better SD: 0.44Equal-weight average — Error correlation rho: 0.9, Average SD minus better SD: 0.72Equal-weight average — Error correlation rho: 1, Average SD minus better SD: 1Error correlation rhoAverage SD minus better SD

Error SDs 10 and 12 as above; below zero the average beats the better member. Check the correlation of the members' backtest errors first: at rho = 0.9 the average has SD 10.7. Equal weights skip estimating weights from short samples; test them on the same origins as the members (Day 23). Computed in Python.

Intermittent demand4 minCroston's two smoothed pieces, the bias that SBA corrects, the library options, and why MAE favours a useless zero forecast.
Croston and SBA for intermittent demand Formula
y^=z^p^,y^SBA=(1−α2)z^p^\hat y = \frac{\hat z}{\hat p}, \qquad \hat y_{\text{SBA}} = \left(1 - \frac{\alpha}{2}\right)\frac{\hat z}{\hat p}
z^\hat z
smoothed size of non-zero demands
p^\hat p
smoothed interval between non-zero demands
1−α/21 - \alpha/2
the Syntetos-Boylan correction for Croston's upward bias
Croston, piece by piece Illustration
012345121416180Sales and the Croston forecasttrue rate 0.625Croston forecastDay (last 80 of 160)Units per day

59 of 80 days sell none. Croston's forecast (blue) changes only after a sale and hovers near the true rate (dashed); a moving average would sag towards zero between sales.

01234567121416180Smoothed size and smoothed gapsize z ≈ 2.45gap p ≈ 4.76DayUnits / days

The two smoothed pieces. Size (orange) moves only when a sale happens; the gap (purple) is the average number of days between sales. The forecast is their ratio.

0.50.60.70.80.050.10.20.30.5Bias grows with alphatrue rate 0.625Croston 0.79SBA 0.59Smoothing constant alphaMean forecast, units/day

Mean forecast over 200 series, by smoothing constant. Croston's ratio drifts above the true rate as alpha grows; multiplying by (1 - alpha/2) (SBA) stays much closer, slightly below at large alpha.

Synthetic: demand on a day with probability 0.25 (think of a spare part at one depot), size 1 + Poisson(1.5), so the true rate is 0.625 per day. Left panels: one series, alpha = 0.1; right: 200 series, days 100-299 of each.

Two smoothed pieces, size and gap; their ratio is biased upwards and SBA scales it down.

Intermittent-demand models in statsforecast 2.0.1 (read from the source) Table
ModelWhat it smoothsKnob that bites
CrostonClassicnon-zero sizes and gaps, SES eachalpha fixed at 0.1, no argument to change it
CrostonOptimizedthe same two piecesalpha fitted per piece; fitted values are slow
CrostonSBACrostonClassic x 0.950.95 = 1 - alpha/2 at alpha 0.1
TSBdemand probability every period, and sizealpha_d and alpha_p have no defaults
ADIDAblock sums, fitted SES, divided back per periodblock length = rounded mean gap
IMAPAthe ADIDA forecast at several block lengths, averagedlengths 1 up to the rounded mean gap

Which one wins on your items is a backtest question, scored on bias and a cost or quantile loss, not on MAE (table below).

An ancillary bought on about 1 flight in 4: true rate 0.625 (hands-on lab) Table
ForecastMean forecastMAERMSE
Zero0.0000.63 (best)1.38 (worst)
Mean so far0.6220.931.24
Croston0.6520.951.25
SBA0.6200.931.25

MAE rewards forecasting zero, which would never stock the item. Judge intermittent methods on bias and a cost or quantile loss.

What a staff-level answer addsadvanced2 minFive nuances that separate a correct interval answer from a senior one.
What a staff-level answer adds Overview
  • Marginal is not conditional

    An 80% band can cover 95% of quiet weekdays and 50% of peaks. Report coverage by slice.

  • Coverage has its own error

    Over n departures its standard error is sqrt(0.8 x 0.2 / n); judge a miss against it.

  • Horizon and label delay

    A 14-day horizon (or a claim that settles in 14 days) makes the newest usable error two weeks old.

  • Price the band

    A stale quantile plan can cost more than planning at the mean; the asymmetry amplifies level errors.

  • Censoring

    Stock-outs and sold-out flights record the cap: their errors understate demand error (Day 28).

Memory anchor

Plan with the quantile the costs ask for, and trust the band only as far as its coverage was checked on errors it had not seen.

Useful for

  • Setting protection levels, safety stock or staffing from a quantile that matches the cost ratio instead of the mean.

  • Auditing a vendor or colleague's "80% interval" with coverage, width and slice checks before it drives an action.

  • Deciding how to handle a hierarchy (chain and stores, region and cells), intermittent spare parts or capped sales without a new model family.

Not suitable for

  • Establishing causal effects of price or capacity - an interval around a forecast is still a forecast (Day 28).

  • Very short series or rare events (a new product, a once-a-decade storm loss) where a calibration set of dozens of errors cannot resolve the quantile you need.

Traps

Reading width as quality

A logistics team celebrates an ETA band that is 20% narrower after a model change, without recomputing coverage. The new band covers 60% of deliveries.

How to avoid it

Report coverage and width together, on held-out data, and compare against a baseline band.

Calibrating on errors you will not have

Claims-reserve residuals from the whole year, including claims that settled after the cutoff, calibrate the band. Offline coverage looks right, live coverage is lower and lags every regime change.

How to avoid it

Build the calibration pool per cutoff from outcomes already known then (settled claims, flown departures), at the planning horizon.

Senior reasoning

A senior asks which action the interval feeds and prices both error directions before choosing a level. They treat coverage as a monitored quantity, not a model property, and expect it to fail first at regime changes, in peak periods and on small segments. They keep the interval method simple enough to explain to the planners who use it, and they never present a band without the evidence that it was checked on data the band had not seen.

Interview question

"Your 90% interval covered only 70% last quarter. What do you do?" Structure the answer.

Recall cards scheduled after completion

  • What are coverage and sharpness, and which one can you not judge alone?
  • How do you choose the planning quantile, and which assumption makes conformal coverage hold?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

References

Optional sections

Deep dive · 15 min

Pinball loss and the quantile. For ρτ(u)=u (τ−1{u<0})\rho_\tau(u)=u\,(\tau-\mathbf 1\{u<0\}) with u=y−q^u=y-\hat q, the expected loss has derivative F(q^)−τF(\hat q)-\tau with respect to q^\hat q, so the minimizer solves F(q^)=τF(\hat q)=\tau. With costs cuc_u (when y>q^y>\hat q) and coc_o (when y<q^y<\hat q), the cost is (cu+co)ρτ(c_u+c_o)\rho_\tau at τ=cu/(cu+co)\tau=c_u/(c_u+c_o), which is why pinball scoring and the newsvendor decision are the same object. In revenue management the same logic sets a protection level for a higher fare class: protect seats until the probability that higher-fare demand exceeds the level equals the fare ratio (the standard single-resource result; Talluri and van Ryzin, 2004, chapter mapping not checked here).

Split conformal in one line. Given nn calibration scores si=∣yi−y^i∣s_i=|y_i-\hat y_i|, the band is y^±s^\hat y\pm\hat s with s^\hat s the ⌈(n+1)(1−α)⌉/n\lceil (n+1)(1-\alpha)\rceil/n empirical quantile of the scores. Under exchangeability of calibration and test points, coverage is at least 1−α1-\alpha and, for continuous scores, at most 1−α+1/(n+1)1-\alpha+1/(n+1). The code in the lesson uses plain empirical quantiles of signed errors, which is the same idea without the finite-sample correction; with 100 errors the correction moves the 0.9 level to about 0.91.

Why time series break it. Exchangeability says that shuffling the calibration and test errors would not change their joint distribution. Seasonality (errors larger in peak weeks), trend and regime shifts make the order informative. Barber et al. (2023) bound the coverage loss for weighted calibration; Gibbs and Candes (2021) update αt+1=αt+γ(α−errt)\alpha_{t+1}=\alpha_t+\gamma(\alpha-\mathrm{err}_t) so that long-run coverage is restored. Stankeviciute et al. (2021) apply conformal calibration to multi-horizon recurrent-network forecasts with a correction across horizons.

Reconciliation. With summing matrix SS and base forecasts y^\hat y, a reconciled forecast is y~=S(S⊤W−1S)−1S⊤W−1y^\tilde y=S(S^\top W^{-1}S)^{-1}S^\top W^{-1}\hat y, where WW is the covariance of base forecast errors (MinT). W=IW=I is the OLS case. Every reconciled set adds up by construction. Reconciliation gains accuracy mainly when different levels carry different information; it does not repair a biased level.

Combinations. The Bates and Granger (1969) result is that a weighted combination can beat each member when the members' errors are not perfectly correlated. Equal weights avoid estimating weights from short samples. FPP3 section 13.4 gives a practical treatment.

Intermittent demand. Croston: smooth the non-zero sizes ztz_t and the gaps ptp_t with exponential smoothing and forecast z^/p^\hat z/\hat p. The ratio of smoothed values overstates the rate (Syntetos and Boylan, 2005); SBA multiplies by 1−α/21-\alpha/2 with α\alpha the smoothing constant. For MAE, the optimal forecast is the median, and a series that is zero on most days has median zero, so MAE rewards the useless forecast of zero. Use a quantile or a scaled squared error, and judge the forecast as a rate for an inventory rule.

Multiple seasonality. Daily airline data carry weekly and annual cycles, plus moving holidays. Options are STL with several seasonal periods (MSTL; Bandara et al.), harmonic regression with Fourier terms (FPP3 12.1), or lag and calendar features in a global model (Day 24). A single seasonal period of 7 misses the annual cycle.

Censored demand. If bookings stop at 169 on a 160-seat aircraft with 6% overbooking, the observed errors at the cap are lower bounds on demand error. Intervals built from them are too narrow on exactly the flights that matter most. Unconstraining methods are surveyed in Weatherford and Polt (2002); Day 28 shows the effect on a what-if.

Hands-on lab: Reconcile a three-route hierarchy and see why MAE rewards a zero forecast · 20 min

Run the code. Part 1 builds independent base forecasts for the network total and three routes (synthetic; top-down is given the true shares, a best case for it). Check that the base forecasts disagree, that bottom-up, top-down and WLS-reconciled forecasts add up, and which level each method improves. Then change sd_base so the total is the most accurate level and see how the ranking changes. Part 2 simulates 300 intermittent series for an ancillary sold on about one flight in four. Compare mean forecasts with the true rate of 0.625 per flight, and compare MAE with RMSE for the zero forecast. Finally write one sentence on which error measure you would use to choose an intermittent-demand method for stocking decisions, and why.

import numpy as np

rng = np.random.default_rng(260)

# ---- 1. A three-route hierarchy: total = ESB + ADB + AYT (synthetic weekly bookings) -------------
S = np.array([[1, 1, 1], [1, 0, 0], [0, 1, 0], [0, 0, 1]], float)  # rows: total, ESB, ADB, AYT
truth = np.array([900.0, 700.0, 450.0])                              # route means
reps = 2000
y = truth + rng.normal(0, [30, 25, 20], (reps, 3))                   # realised route demand
actual = y @ S.T                                                     # columns: total, ESB, ADB, AYT
sd_base = np.array([55.0, 40.0, 30.0, 25.0])                         # base-forecast error sd per level
base = actual + rng.normal(0, sd_base, actual.shape)                 # independent base forecasts per level

w_inv = np.diag(1 / sd_base**2)                                      # diagonal W: weight by error variance
P = np.linalg.solve(S.T @ w_inv @ S, S.T @ w_inv)                    # (S'W^-1 S)^-1 S'W^-1
reconciled = base @ (S @ P).T                                        # y_tilde = S P y_hat
bottom_up = np.column_stack([base[:, 1:].sum(1), base[:, 1:]])
top_down = np.column_stack([base[:, 0], base[:, [0]] * (truth / truth.sum())])
print("method          RMSE total  RMSE routes  mean |total - sum(routes)|")
for name, f in [("base", base), ("bottom-up", bottom_up), ("top-down", top_down), ("WLS reconciled", reconciled)]:
    err = f - actual
    gap = np.abs(f[:, 0] - f[:, 1:].sum(1)).mean()
    print(f"{name:15s} {np.sqrt((err[:, 0] ** 2).mean()):9.1f} {np.sqrt((err[:, 1:] ** 2).mean()):11.1f} {gap:12.1f}")

# ---- 2. Intermittent demand: an ancillary bought on about 1 flight in 4 ---------------------------
def croston(d, alpha, sba=False):
    nz = np.flatnonzero(d)
    size, gap, last = d[nz[0]], nz[0] + 1, nz[0]
    out = np.zeros(len(d))
    for t in range(len(d)):
        out[t] = size / gap * (1 - alpha / 2 if sba else 1)          # forecast made before seeing d[t]
        if d[t] > 0:
            size += alpha * (d[t] - size)
            gap += alpha * ((t - last) - gap)
            last = t
    return out

p, T, burn = 0.25, 300, 100
rate = p * 2.5                                                       # E[size] = 1 + 1.5
rows = {"zero": [], "mean so far": [], "croston": [], "sba": []}
for _ in range(300):
    d = rng.binomial(1, p, T) * (1 + rng.poisson(1.5, T))
    if d.sum() == 0:
        continue
    fc = {"zero": np.zeros(T), "mean so far": np.array([d[:t].mean() if t else 0.0 for t in range(T)]),
          "croston": croston(d, 0.1), "sba": croston(d, 0.1, True)}
    for k, f in fc.items():
        e = d[burn:] - f[burn:]
        rows[k].append((f[burn:].mean(), np.abs(e).mean(), np.sqrt((e ** 2).mean())))
print(f"\ntrue demand rate {rate:.3f} per flight")
for k, v in rows.items():
    m, mae, rmse = np.mean(v, axis=0)
    print(f"{k:12s} mean forecast {m:.3f}  MAE {mae:.2f}  RMSE {rmse:.2f}")
Interview practice · 10 min

A logistics planner asks for "a 95% range" on next month's parcel volume at a sorting hub and wants to contract trucks and temporary staff at the top of it. How do you respond?

Content version 6 · last verified 2026-10-18 · depends on numpy, scipy