Advanced Forecasting
By the end: From a set of forecast errors you build a band, check its coverage and width using only errors that existed at the cutoff, choose the planning quantile from a cost ratio, and say which assumption a stale calibration violates.
At a glance
A forecast band is a promise about coverage; choose the planning quantile from the costs and recheck the promise wherever the regime can move.
Three objects
Confidence interval for a mean, prediction interval for one outcome, quantile for a decision.
Coverage and sharpness
Calibrated first, then as narrow as possible; read the two together.
Quantile from costs
tau* = c_under / (c_under + c_over) sets stock, staff or a reserve; score it with pinball loss.
Conformal
Past errors set the width; the guarantee needs exchangeable errors.
Monitor coverage
A shift drops coverage first; recalibrate on errors known at each cutoff.
Recall
Without notes: (1) at a D-14 cutoff for departure date T, which earlier departures have a known forecast error? (2) Which point forecast minimizes MAE, and which loss has the 0.9 quantile as its minimizer? Then clear your due review cards.
Concept
A point forecast of "100 more bookings" is one number; the decision needs the spread and the side that hurts more. Judge a band on two things that pull apart: coverage (an "80%" band that caught 43% is miscalibrated) and sharpness (an enormous band always covers). Aim for the sharpest band that is still calibrated.
Predict first
A band is rebuilt every week from the last 100 errors. A competitor adds capacity today. What happens to coverage over the next two weeks?
Text description of the figure
A simulated series with controls for confidence level, noise regime (constant, heteroscedastic, regime change) and horizon, and a selector for the method (Gaussian, empirical residuals, split conformal). The band and the actuals inside and outside it are drawn; a table reports coverage and mean width for all three methods, and a rolling coverage chart shows it falling away from the nominal level.
| Criterion | Confidence interval | Prediction interval | Quantile forecast |
|---|---|---|---|
| Covers | An unknown mean or parameter (Day 4) | One future outcome, including its own noise | The value outcomes stay below with probability tau |
| Narrows with more data? | Yes, towards zero width | Only down to the noise level | Converges to the true quantile |
| Example | Retail: the average Friday basket value across all stores | Energy: tomorrow's peak load at one substation | Insurance: the 0.9 quantile of next quarter's claims, to reserve |
An 80% prediction interval is the pair of quantiles at 0.1 and 0.9. The airline code below plans at the 0.27 quantile of pickup.
Honest but vague
Calibrated and wide: safe, costly to plan with.
The goal
Calibrated and as narrow as the data allow.
Off-centre
Wide and still missing: the forecast itself is biased.
Confidently wrong
Narrow and miscalibrated: the most dangerous band.
Maximize sharpness subject to calibration (Gneiting, Balabdaoui and Raftery, 2007).
- cost per seat of planning too low: dilution, EUR 30 (hypothetical)
- cost per seat of planning too high: spoilage, EUR 80 (hypothetical)
- the quantile that minimizes expected cost; 0.5 only when the costs are equal
The same rule sets a nurse roster (understaffing dearer than overstaffing) or an energy trader's bid; only c_u and c_o change. The critical ratio is only as reliable as those two unit costs.
Simulated pickup outcomes. Planning at the median leaves as much room above as below; with spoilage the dearer error, the cheapest plan sits low, at the 0.27 quantile.
Expected cost per departure for every planning quantile. The minimum lands at 0.28 (theory: 0.27); planning at the median costs EUR 84 more per departure here.
Change the cost ratio and the best quantile moves along tau* = r / (1 + r); the saving over planning at the median grows quickly once one error costs several times the other.
Synthetic: 4,000 pickup outcomes, mean 100 and sd 12 seats (seeded). Hypothetical costs EUR 30 (too low) and EUR 80 (too high) per seat; the saving curve is relative to planning at the median.
Expected cost computed for every planning quantile on simulated outcomes; the minimum sits where the cost ratio says.
- Gaussian band
- adaptive conformal band (dashed)
- outcome outside the Gaussian band
Before the break the band (shaded) catches about 8 in 10 outcomes. After it, the noise is 2.6 times larger and the band is still sized from old errors: misses (red) pile up until the window refills.
Trailing 30-step coverage; dashed line: the nominal 80%. All three sag after the break; the adaptive band reacts within a few misses because every miss lowers its target miscoverage.
After the break: coverage over all later steps and over the first 40, with mean width. Coverage is bought with width; the adaptive band pays more of it, sooner.
Synthetic local-level series (seed 26), noise sd 1 then 2.6 from step 186; SES one-step forecasts; 80% bands from the last 100 errors (interval-lab engine). Adaptive conformal uses gamma = 0.05.
Rolling bands keep the old width until the window refills; adaptive conformal reacts to its own misses, recovers coverage sooner and pays with a wider band.
Leaning towards one side amplifies a level error, so a stale quantile plan is worse than ignoring the asymmetry.
- Three objects: a confidence interval (a mean), a prediction interval (one future departure), a quantile forecast (the value outcomes stay below with probability tau).
- Quantile from costs: the newsvendor critical ratio, scored with pinball loss.
- Band from past errors: conformal prediction gives marginal coverage under Exchangeability; shifts break it.
- More pieces: reconciliation, combinations, intermittent demand.
Assumptions
Calibration errors are exchangeable with future errors. Seasonality, drift and breaks violate this, and the guarantee is marginal, averaged over departures, not per route or month.
Calibration errors existed at the cutoff (at D-14, only departures that have flown) and match the planning horizon.
The unit costs are known; the critical ratio is only as reliable as c_under and c_over (hypothetical here).
Application and workplace decision
Run the simulation (synthetic). Predict which band keeps 80% coverage after the shift.
- Pickup follows a weekday pattern; the forecast is the weekday mean of departures 0-199.
- Departures 200-299 calibrate the bands; 300-599 are the test, with pickup up 18% from departure 450 when a competitor leaves.
- Forecasts are issued at D-14, so rolling calibration may use only departures that have flown.
- Three 80% bands (Gaussian, static empirical, rolling empirical), and the cost of planning at the mean versus the cost quantile (hypothetical EUR 30 dilution, EUR 80 spoilage per seat).
Python
import numpy as np
from scipy.stats import norm
rng = np.random.default_rng(26)
n, fit, test, shift, lag = 600, 200, 300, 450, 14 # departures in date order; forecasts issued at D-14
mu = 100 + 12 * np.sin(2 * np.pi * np.arange(n) / 7) # weekday pickup level, seats (synthetic)
mu[shift:] *= 1.18 # a competitor exits at departure 450: pickup +18%
y = rng.normal(mu, 0.12 * mu) # realised pickup per departure
f = np.array([y[:fit][np.arange(fit) % 7 == d % 7].mean() for d in range(n)]) # weekday means, fit period
e = y - f # signed forecast error, seats
c_under, c_over = 30, 80 # hypothetical EUR per seat: dilution vs spoilage
q_star = c_under / (c_under + c_over) # newsvendor critical ratio = 0.27
def quantiles(i, method): # error quantiles usable at departure i's cutoff
ps = [0.1, 0.9, q_star]
if method == "gaussian":
return norm.ppf(ps) * e[:fit].std()
pool = e[fit:test] if method == "static" else e[max(fit, i - lag - 99):i - lag + 1] # flown only
return np.quantile(pool, ps)
print("method period coverage80 width cost@mean cost@q* (EUR per departure)")
for m in ["gaussian", "static", "rolling"]:
for name, idx in [("before", np.arange(test, shift)), ("after", np.arange(shift, n))]:
q = np.array([quantiles(i, m) for i in idx])
inside = (y[idx] >= f[idx] + q[:, 0]) & (y[idx] <= f[idx] + q[:, 1])
cost = lambda plan: np.where(y[idx] > plan, c_under * (y[idx] - plan), c_over * (plan - y[idx])).mean()
print(f"{m:9s} {name:6s} {inside.mean():.2f} {np.mean(q[:, 1] - q[:, 0]):6.1f} {cost(f[idx]):8.0f} {cost(f[idx] + q[:, 2]):7.0f}")
Mini-assessment
Technical depth
The quantile, derived from the costsThe pinball loss defined, its minimizer derived step by step, a bakery example worked by hand, how much data a tail quantile needs, and the library defaults that bite.
- the tau-quantile: the smallest q with F(q) >= tau; an 80% prediction interval is the pair q_0.1, q_0.9
- the quantile forecast (the plan)
- the target quantile; the loss is minimized in expectation by the true tau-quantile
- half the absolute error, so the median minimizes MAE (Gneiting, 2011)
At tau* the money cost of a plan equals (c_u + c_o) times its pinball loss.
Nudge the plan up
Raising q by a small dq adds (1 - tau) dq on the share F(q) of outcomes below q, saves tau dq on the 1 - F(q) above.
The slope
Net change per unit: dE[L]/dq = (1 - tau) F(q) - tau (1 - F(q)) = F(q) - tau.
Set it to zero
The expected loss is flat exactly where F(q*) = tau: the minimizer is the tau-quantile, whatever the distribution's shape.
Costs choose tau
With unit costs, expected money cost = (c_u + c_o) x pinball loss at tau = c_u / (c_u + c_o): the score and the decision agree.
The same argument with c_u and c_o in place of tau and 1 - tau is the newsvendor result; the worked table below pushes real numbers through it.
| Plan (units) | Expected under-cost | Expected over-cost | Total per day |
|---|---|---|---|
| 9 | 2 x 6/5 = 2.40 | 1 x 1/5 = 0.20 | 2.60 |
| 10 | 2 x 3/5 = 1.20 | 1 x 3/5 = 0.60 | 1.80 |
| 11 | 2 x 1/5 = 0.40 | 1 x 6/5 = 1.20 | 1.60 (minimum) |
| 12 | 0 | 1 x 10/5 = 2.00 | 2.00 |
Demand 8, 9, 10, 11 or 12 units, equally likely; c_u = EUR 2 lost margin per unit short, c_o = EUR 1 markdown per unit over. tau* = 2/(2+1) = 2/3, and the 2/3 quantile of demand is 11: exactly where the hand-computed total is lowest (verified in Python).
Bar = standard deviation of the sample quantile across 200,000 simulated pools of 30 normal errors (in units of the error SD); note = how many errors sit above it on average. The estimate is consistent, but the further into the tail, the more errors it needs. Computed in Python.
| Tool | How to ask for a quantile or band | Default that bites |
|---|---|---|
| scikit-learn 1.9 QuantileRegressor | quantile=tau (default 0.5) | alpha=1.0 L1 penalty shrinks coefficients; set alpha=0 for a plain linear quantile |
| scikit-learn HistGradientBoostingRegressor | loss='quantile', quantile=tau | default loss is squared error: no quantile unless asked |
| scikit-learn GradientBoostingRegressor | loss='quantile', alpha=tau | alpha defaults to 0.9: the name means the quantile here |
| statsforecast 2.0 ConformalIntervals | prediction_intervals=ConformalIntervals(h, n_windows) | n_windows=2, h=1 by default: set h to your horizon |
Read from the installed packages (inspect.signature and the source). Separately fitted quantile models can cross; sort or monotonize before use.
Conformal bands and their one assumptionadvancedSplit conformal worked by hand on ten residuals, the rank argument behind its guarantee drawn and simulated, and the adaptive update for shifts.
| Step | Rule | With the ten residuals |
|---|---|---|
| 1 Collect | Absolute errors on calibration rows the model never saw | 1.2, 1.9, 2.4, 3.0, 3.3, 4.1, 4.9, 5.6, 7.0, 9.8 (sorted, seats) |
| 2 Rank | k = ceil((n+1)(1-alpha)) | k = ceil(11 x 0.8) = ceil(8.8) = 9 |
| 3 Width | q-hat = the k-th smallest score | q-hat = 7.0 seats (the 9th value) |
| 4 Band | forecast +/- q-hat | a forecast of 120 becomes 113 to 127 |
| 5 Guarantee | coverage in [1-alpha, 1-alpha + 1/(n+1)] | between 80.0% and 89.1%, if the errors are exchangeable |
The +1 in the rank is the finite-sample correction: with only 10 errors the 80% band uses the 9th largest score, wider than the plain 0.8 quantile. Rank and bounds verified in Python; the same five steps price a claims reserve or an ETA band as well as a seat forecast.
Ten scores cut the line into 11 slots. Exchangeable means the new error is just as likely to be the smallest as the largest, so each slot has probability 1/11; the band q-hat = 7.0 covers the first 9: 9/11 = 81.8%, between 80% and 80% + 1/11. The slot widths in seats do not matter, only the ranks.
20,000 simulated calibration pools per row. The rank argument holds for any error distribution and any n; it fails as soon as the new errors come from a different regime.
Simulation: |standard normal| calibration errors, k = ceil((n + 1) x 0.8); the shifted row draws the new error with 1.5 times the spread, so "rank theory" no longer applies to it. Seeded, 20,000 trials per row.
The ten residuals of the worked example, sorted. Under exchangeability every gap is equally likely; the band covers when the new error lands below the 9th. Simulated check below, with and without a shift. Seasonality, trend and regime shifts break the equal-gap argument (Angelopoulos and Bates, 2023).
- forecast errors on n calibration rows
- miscoverage: 0.2 for an 80% band
The guarantee is marginal, averaged over departures, not per route or per month. Weighted and adaptive variants exist for shifts (Barber et al., 2023; Gibbs and Candes, 2021).
- target miscoverage, 0.2 for an 80% band
- the level actually used at step t; a miss lowers it, so the next band widens
- step size: larger reacts faster and makes the width jump after every miss; choose it on a backtest that contains a past shift
Gibbs and Candes (2021) bound the long-run miscoverage gap for any sequence of outcomes; it says nothing about coverage in a given week.
Monitoring coverage as a binomialCoverage is a proportion with a computable standard error; today's before-and-after numbers, the monitoring loop, and a diagnosis path.
- outcomes inside the band out of n; each departure, order or claim is one draw
- SE = 0.057: an observed 0.68 sits 2.1 SE below 0.8, a real miss, not noise
- SE = 0.115: the same 0.68 is within about one SE of 0.8 and proves nothing yet
Judge every coverage number against this band before acting, with draws from the band's own period and segment. Worked values computed in Python; nearby outcomes share shocks, so treat the SE as a lower bound.
150 test departures in each period; pickup rises 18% from departure 450 (synthetic).
- Pool errors known at the cutoffat D-14: flown departures only, at the planning horizonrecalibrate ↓
- Rebuild the band on a windowlonger window: less noise, slower reactionscore ↓
- Report coverage and widthby month, market and segment, never one alonecompare ↓
- 30-day coverage in 70-90%?outside: alert, then diagnose (next figure)next cutoff ↓
- ↺ back to 1 · Pool errors known at the cutoff
Each step repairs one way a band goes stale. The window length is the one dial; it trades reaction speed after a shift against noise in quiet periods.
- Misses on one side?
- yes: to Biased point forecast
no - Since a known date?
- yes: to Exchangeability broke
no - Only in some slices?
- yes: to Marginal ≠ conditional
no - Check horizon and poolhorizon errors known at the cutoff
- Biased point forecastfix the level first; the band only follows
- Exchangeability brokerecalibrate on recent errors, or ACI
- Marginal ≠ conditionalcalibrate per segment or use CQR
Read coverage with its sampling error first, sqrt(0.8 x 0.2 / n) for an 80% band; at n = 100 that is 0.04.
Reconciliation is one projectionadvancedWhy levels disagree, the projection behind every fix with a three-number example worked by hand, a computed before-and-after, and what each method costs.
Each dot is one period: the total's own forecast against the sum of the three route forecasts. Off the diagonal they disagree, by 11.1 on average; every reconciled set lies exactly on it.
- Base (independent)
- Bottom-up
- Top-down
- MinT
Error by level (RMSE) when the total's base forecast has error sd 10 and routes have sd 8, 6 and 4. Bottom-up inherits all route noise in the total; MinT improves every level. Top-down wins on routes only because the simulated shares never move.
Turn one knob: how noisy the total's own forecast is. Bottom-up ignores it; the base total gets worse and worse; MinT tracks the better of the two because it weights the levels by their estimated error covariance.
Synthetic hierarchy (total = A + B + C, levels 120/80/50 with a shared season), 60 training and 80 test periods, seed 26. Reconciliation-lab engine: MinT with a shrinkage covariance; top-down uses average historical shares.
Base forecasts disagree across levels; MinT makes them add up and follows whichever level is more accurate.
- summing matrix: rows are total and routes, columns are routes
- base forecasts at every level
- covariance of base forecast errors; W = I gives OLS, a diagonal W gives WLS
- bottom-up keeps only routes; top-down splits the total by shares
- a projection: any base vector maps to a coherent one, coherent ones pass unchanged; OLS takes the nearest coherent point, MinT measures 'near' with W
Because the result is S times something, every reconciled set adds up. MinT minimizes the trace of the reconciled error covariance (Wickramasuriya et al., 2019).
| Series | Base forecast | OLS reconciled | What moved |
|---|---|---|---|
| Chain total | 100 | 96.67 | down by gap/3 |
| Store A | 40 | 43.33 | up by gap/3 |
| Store B | 50 | 53.33 | up by gap/3 |
| Coherence | gap = 100 - 90 = +10 | 43.33 + 53.33 = 96.67 | the 10-unit gap is split equally |
G = (S'S)^-1 S' projects the base forecasts onto the coherent subspace: the closest adding-up set in least squares, so each series concedes a third of the gap. A non-identity W (WLS, MinT) leans the split towards the levels with smaller error variance. Verified in Python.
| Criterion | Bottom-up | Top-down | WLS / MinT |
|---|---|---|---|
| Uses | Route forecasts only | Total plus historical shares | Every level, weighted by error variance |
| The total | Sums every route's noise | Stable: the total's own forecast | Leans on the more accurate level |
| The routes | Keeps full route detail | Breaks when shares go stale | Keeps detail, pulled towards the total |
| Unbiased bases stay unbiased? | Yes | No, once shares move | Yes |
| Hands-on lab RMSE, total / routes | 55.3 / 32.2 | 55.0 / 27.8 (given the true shares) | 38.5 / 28.8 |
Hands-on lab, synthetic three-route hierarchy: independent base forecasts score 55.0 / 32.2 and miss adding up by 62.5 on average; every method here closes that gap to 0. Top-down is handed the true shares, its best case. None of them repairs a biased level: check bias per level first.
Four ways to build a bandWhat Gaussian, split conformal, CQR and adaptive conformal each assume, and which one your backtest errors call for.
| Criterion | Gaussian on residuals | Split conformal | Conformalized quantile regression | Adaptive conformal (ACI) |
|---|---|---|---|---|
| Assumes | Normal, constant-variance errors | Exchangeable calibration and test errors | Exchangeability; quantile models for the shape | Long-run coverage only, no exchangeability |
| Width adapts to the row? | No, one width for every row | No, unless scores are scaled | Yes: wider where the model expects more spread | No, it moves one level for all rows |
| Reacts to a shift | Only as fast as the residual window | Only as fast as the calibration window | Same as split conformal | Yes: every miss lowers the target miscoverage |
| Cost | One standard deviation | A held-out calibration block | Two quantile models plus calibration | One step size gamma to choose |
Plain empirical quantiles of past errors are split conformal without the +1 rank correction. CQR: Romano, Patterson and Candes (2019). ACI: Gibbs and Candes (2021). The guarantee of each is marginal, not per route or month.
- Backtest errorsat the horizon: shape, slices, breaks
- stable, normalGaussian bandconformal adds little here
- otherwiseSpread varies by slice?segment, season, route
- yesConformal quantilesCQR: wider where spread is larger
- noRegime can move fast?
- yesAdaptive conformalACI: long-run coverage, jumpier width
- noSplit conformalrolling calibration window
Conformal earns its keep when you cannot trust a distribution, not when errors are already normal and stable. The hospital case (quiet weekdays, winter peaks) is the classic reason for a width that adapts to the row.
Forecast combinations, with the variance algebraThe error variance of an average derived, a two-forecast example worked by hand, and the correlation at which averaging stops paying.
- errors of two unbiased forecasts, e.g. ETS and a GBM for a telecom cell's daily traffic
- their error SDs (illustrative); rho = 0.5 is the error correlation
- (100 + 144 + 120) / 4 = 91, SD 9.54: below the better member's 10
- the variance-minimizing weight on forecast 1: 84 / 124 = 0.68, SD 9.33, barely better than 9.54
The gain comes from rho < 1; at rho = 1 averaging cannot help. Estimated weights carry their own noise, which is why equal weights often hold up on short samples (Bates and Granger, 1969). Numbers checked in Python.
Error SDs 10 and 12 as above; below zero the average beats the better member. Check the correlation of the members' backtest errors first: at rho = 0.9 the average has SD 10.7. Equal weights skip estimating weights from short samples; test them on the same origins as the members (Day 23). Computed in Python.
Intermittent demandCroston's two smoothed pieces, the bias that SBA corrects, the library options, and why MAE favours a useless zero forecast.
- smoothed size of non-zero demands
- smoothed interval between non-zero demands
- the Syntetos-Boylan correction for Croston's upward bias
59 of 80 days sell none. Croston's forecast (blue) changes only after a sale and hovers near the true rate (dashed); a moving average would sag towards zero between sales.
The two smoothed pieces. Size (orange) moves only when a sale happens; the gap (purple) is the average number of days between sales. The forecast is their ratio.
Mean forecast over 200 series, by smoothing constant. Croston's ratio drifts above the true rate as alpha grows; multiplying by (1 - alpha/2) (SBA) stays much closer, slightly below at large alpha.
Synthetic: demand on a day with probability 0.25 (think of a spare part at one depot), size 1 + Poisson(1.5), so the true rate is 0.625 per day. Left panels: one series, alpha = 0.1; right: 200 series, days 100-299 of each.
Two smoothed pieces, size and gap; their ratio is biased upwards and SBA scales it down.
| Model | What it smooths | Knob that bites |
|---|---|---|
| CrostonClassic | non-zero sizes and gaps, SES each | alpha fixed at 0.1, no argument to change it |
| CrostonOptimized | the same two pieces | alpha fitted per piece; fitted values are slow |
| CrostonSBA | CrostonClassic x 0.95 | 0.95 = 1 - alpha/2 at alpha 0.1 |
| TSB | demand probability every period, and size | alpha_d and alpha_p have no defaults |
| ADIDA | block sums, fitted SES, divided back per period | block length = rounded mean gap |
| IMAPA | the ADIDA forecast at several block lengths, averaged | lengths 1 up to the rounded mean gap |
Which one wins on your items is a backtest question, scored on bias and a cost or quantile loss, not on MAE (table below).
| Forecast | Mean forecast | MAE | RMSE |
|---|---|---|---|
| Zero | 0.000 | 0.63 (best) | 1.38 (worst) |
| Mean so far | 0.622 | 0.93 | 1.24 |
| Croston | 0.652 | 0.95 | 1.25 |
| SBA | 0.620 | 0.93 | 1.25 |
MAE rewards forecasting zero, which would never stock the item. Judge intermittent methods on bias and a cost or quantile loss.
What a staff-level answer addsadvancedFive nuances that separate a correct interval answer from a senior one.
Marginal is not conditional
An 80% band can cover 95% of quiet weekdays and 50% of peaks. Report coverage by slice.
Coverage has its own error
Over n departures its standard error is sqrt(0.8 x 0.2 / n); judge a miss against it.
Horizon and label delay
A 14-day horizon (or a claim that settles in 14 days) makes the newest usable error two weeks old.
Price the band
A stale quantile plan can cost more than planning at the mean; the asymmetry amplifies level errors.
Censoring
Stock-outs and sold-out flights record the cap: their errors understate demand error (Day 28).
Memory anchor
Plan with the quantile the costs ask for, and trust the band only as far as its coverage was checked on errors it had not seen.
Useful for
Setting protection levels, safety stock or staffing from a quantile that matches the cost ratio instead of the mean.
Auditing a vendor or colleague's "80% interval" with coverage, width and slice checks before it drives an action.
Deciding how to handle a hierarchy (chain and stores, region and cells), intermittent spare parts or capped sales without a new model family.
Not suitable for
Establishing causal effects of price or capacity - an interval around a forecast is still a forecast (Day 28).
Very short series or rare events (a new product, a once-a-decade storm loss) where a calibration set of dozens of errors cannot resolve the quantile you need.
Traps
Reading width as quality
A logistics team celebrates an ETA band that is 20% narrower after a model change, without recomputing coverage. The new band covers 60% of deliveries.
How to avoid it
Report coverage and width together, on held-out data, and compare against a baseline band.
Calibrating on errors you will not have
Claims-reserve residuals from the whole year, including claims that settled after the cutoff, calibrate the band. Offline coverage looks right, live coverage is lower and lags every regime change.
How to avoid it
Build the calibration pool per cutoff from outcomes already known then (settled claims, flown departures), at the planning horizon.
Senior reasoning
A senior asks which action the interval feeds and prices both error directions before choosing a level. They treat coverage as a monitored quantity, not a model property, and expect it to fail first at regime changes, in peak periods and on small segments. They keep the interval method simple enough to explain to the planners who use it, and they never present a band without the evidence that it was checked on data the band had not seen.
Interview question
"Your 90% interval covered only 70% last quarter. What do you do?" Structure the answer.
Recall cards scheduled after completion
- What are coverage and sharpness, and which one can you not judge alone?
- How do you choose the planning quantile, and which assumption makes conformal coverage hold?
Prerequisites and related concepts
- Confidence intervals · day 4
- Forecast baselines · day 23
- Rolling-origin backtesting · day 25
- Forecast metrics · day 16
- Cost of being wrong · day 1
- Prediction intervals and quantiles
- Coverage
- Conformal prediction
- Hierarchical forecasting and reconciliation
- Forecast combinations
- Intermittent demand
- Forecast decision economics
- Per-horizon error and stability · day 25
- Capacity-censored demand · day 28
- Forecasting vs causal what-if · day 5
- Business vs ML metric · day 16
References
- Probabilistic Forecasts, Calibration and Sharpnesscore
The paradigm of maximizing sharpness subject to calibration, and calibration diagnostics for probabilistic forecasts.
- Conformal Prediction: A Gentle Introductioncore
Split conformal prediction with a calibration set, finite-sample marginal coverage and the exchangeability condition. (Introductory chapters on split conformal prediction and its guarantee)
- Quantiles as optimal point forecastsoptional
Quantiles are the optimal point forecasts under asymmetric piecewise linear loss, with the level set by the costs of under- and over-prediction.
- Conformal prediction beyond exchangeabilityoptional
Weighted conformal methods when exchangeability fails, for example under drift.
- Adaptive Conformal Inference Under Distribution Shiftoptional
Online adjustment of the miscoverage level to keep long-run coverage under distribution shift.
- Forecasting: Principles and Practice (3rd edition)optional
Distributional forecasts and the quantile (pinball) score, hierarchical reconciliation, complex seasonality, counts and intermittent demand, and combinations. (Sections 5.5 and 5.9; 11.2 and 11.3; 12.1; 13.2 and 13.4)
- Optimal Forecast Reconciliation for Hierarchical and Grouped Time Series Through Trace Minimizationoptional
MinT reconciliation minimizes the trace of the reconciled error covariance matrix.
- The accuracy of intermittent demand estimatesoptional
Croston's estimator is biased and the Syntetos-Boylan approximation corrects it.
- Conformal Time-series Forecastingoptional
Conformal intervals for multi-horizon forecasts, with a correction across horizons, as a time-series application.
- Conformalized Quantile Regressionoptional
Conformalized quantile regression, which calibrates quantile-model bands so that their width can adapt to the row.
- Forecasting and Stock Control for Intermittent Demandsoptional
Croston's method smooths non-zero demand sizes and the intervals between them separately for intermittent series.
- The Combination of Forecastsoptional
Founding result that combined forecasts can beat the individual forecasts that go into them.
- MSTL: A Seasonal-Trend Decomposition Algorithm for Time Series with Multiple Seasonal Patternsoptional
STL extended to several seasonal periods by iterative decomposition, for the multiple-seasonality note.
- Better unconstraining of airline demand data in revenue management systems for improved forecast accuracy and greater revenuesoptional
Methods for unconstraining capacity-limited airline booking data; supports the claim that bookings on closed flights understate demand.
- The Theory and Practice of Revenue Managementoptional
Standard revenue-management reference for protection levels, which apply the same quantile rule (chapter mapping not checked here).
Optional sections
Deep dive
Pinball loss and the quantile. For with , the expected loss has derivative with respect to , so the minimizer solves . With costs (when ) and (when ), the cost is at , which is why pinball scoring and the newsvendor decision are the same object. In revenue management the same logic sets a protection level for a higher fare class: protect seats until the probability that higher-fare demand exceeds the level equals the fare ratio (the standard single-resource result; Talluri and van Ryzin, 2004, chapter mapping not checked here).
Split conformal in one line. Given calibration scores , the band is with the empirical quantile of the scores. Under exchangeability of calibration and test points, coverage is at least and, for continuous scores, at most . The code in the lesson uses plain empirical quantiles of signed errors, which is the same idea without the finite-sample correction; with 100 errors the correction moves the 0.9 level to about 0.91.
Why time series break it. Exchangeability says that shuffling the calibration and test errors would not change their joint distribution. Seasonality (errors larger in peak weeks), trend and regime shifts make the order informative. Barber et al. (2023) bound the coverage loss for weighted calibration; Gibbs and Candes (2021) update so that long-run coverage is restored. Stankeviciute et al. (2021) apply conformal calibration to multi-horizon recurrent-network forecasts with a correction across horizons.
Reconciliation. With summing matrix and base forecasts , a reconciled forecast is , where is the covariance of base forecast errors (MinT). is the OLS case. Every reconciled set adds up by construction. Reconciliation gains accuracy mainly when different levels carry different information; it does not repair a biased level.
Combinations. The Bates and Granger (1969) result is that a weighted combination can beat each member when the members' errors are not perfectly correlated. Equal weights avoid estimating weights from short samples. FPP3 section 13.4 gives a practical treatment.
Intermittent demand. Croston: smooth the non-zero sizes and the gaps with exponential smoothing and forecast . The ratio of smoothed values overstates the rate (Syntetos and Boylan, 2005); SBA multiplies by with the smoothing constant. For MAE, the optimal forecast is the median, and a series that is zero on most days has median zero, so MAE rewards the useless forecast of zero. Use a quantile or a scaled squared error, and judge the forecast as a rate for an inventory rule.
Multiple seasonality. Daily airline data carry weekly and annual cycles, plus moving holidays. Options are STL with several seasonal periods (MSTL; Bandara et al.), harmonic regression with Fourier terms (FPP3 12.1), or lag and calendar features in a global model (Day 24). A single seasonal period of 7 misses the annual cycle.
Censored demand. If bookings stop at 169 on a 160-seat aircraft with 6% overbooking, the observed errors at the cap are lower bounds on demand error. Intervals built from them are too narrow on exactly the flights that matter most. Unconstraining methods are surveyed in Weatherford and Polt (2002); Day 28 shows the effect on a what-if.
Hands-on lab: Reconcile a three-route hierarchy and see why MAE rewards a zero forecast
Run the code. Part 1 builds independent base forecasts for the network total and three routes (synthetic; top-down is given the true shares, a best case for it). Check that the base forecasts disagree, that bottom-up, top-down and WLS-reconciled forecasts add up, and which level each method improves. Then change sd_base so the total is the most accurate level and see how the ranking changes. Part 2 simulates 300 intermittent series for an ancillary sold on about one flight in four. Compare mean forecasts with the true rate of 0.625 per flight, and compare MAE with RMSE for the zero forecast. Finally write one sentence on which error measure you would use to choose an intermittent-demand method for stocking decisions, and why.
import numpy as np
rng = np.random.default_rng(260)
# ---- 1. A three-route hierarchy: total = ESB + ADB + AYT (synthetic weekly bookings) -------------
S = np.array([[1, 1, 1], [1, 0, 0], [0, 1, 0], [0, 0, 1]], float) # rows: total, ESB, ADB, AYT
truth = np.array([900.0, 700.0, 450.0]) # route means
reps = 2000
y = truth + rng.normal(0, [30, 25, 20], (reps, 3)) # realised route demand
actual = y @ S.T # columns: total, ESB, ADB, AYT
sd_base = np.array([55.0, 40.0, 30.0, 25.0]) # base-forecast error sd per level
base = actual + rng.normal(0, sd_base, actual.shape) # independent base forecasts per level
w_inv = np.diag(1 / sd_base**2) # diagonal W: weight by error variance
P = np.linalg.solve(S.T @ w_inv @ S, S.T @ w_inv) # (S'W^-1 S)^-1 S'W^-1
reconciled = base @ (S @ P).T # y_tilde = S P y_hat
bottom_up = np.column_stack([base[:, 1:].sum(1), base[:, 1:]])
top_down = np.column_stack([base[:, 0], base[:, [0]] * (truth / truth.sum())])
print("method RMSE total RMSE routes mean |total - sum(routes)|")
for name, f in [("base", base), ("bottom-up", bottom_up), ("top-down", top_down), ("WLS reconciled", reconciled)]:
err = f - actual
gap = np.abs(f[:, 0] - f[:, 1:].sum(1)).mean()
print(f"{name:15s} {np.sqrt((err[:, 0] ** 2).mean()):9.1f} {np.sqrt((err[:, 1:] ** 2).mean()):11.1f} {gap:12.1f}")
# ---- 2. Intermittent demand: an ancillary bought on about 1 flight in 4 ---------------------------
def croston(d, alpha, sba=False):
nz = np.flatnonzero(d)
size, gap, last = d[nz[0]], nz[0] + 1, nz[0]
out = np.zeros(len(d))
for t in range(len(d)):
out[t] = size / gap * (1 - alpha / 2 if sba else 1) # forecast made before seeing d[t]
if d[t] > 0:
size += alpha * (d[t] - size)
gap += alpha * ((t - last) - gap)
last = t
return out
p, T, burn = 0.25, 300, 100
rate = p * 2.5 # E[size] = 1 + 1.5
rows = {"zero": [], "mean so far": [], "croston": [], "sba": []}
for _ in range(300):
d = rng.binomial(1, p, T) * (1 + rng.poisson(1.5, T))
if d.sum() == 0:
continue
fc = {"zero": np.zeros(T), "mean so far": np.array([d[:t].mean() if t else 0.0 for t in range(T)]),
"croston": croston(d, 0.1), "sba": croston(d, 0.1, True)}
for k, f in fc.items():
e = d[burn:] - f[burn:]
rows[k].append((f[burn:].mean(), np.abs(e).mean(), np.sqrt((e ** 2).mean())))
print(f"\ntrue demand rate {rate:.3f} per flight")
for k, v in rows.items():
m, mae, rmse = np.mean(v, axis=0)
print(f"{k:12s} mean forecast {m:.3f} MAE {mae:.2f} RMSE {rmse:.2f}")
Interview practice
A logistics planner asks for "a 95% range" on next month's parcel volume at a sorting hub and wants to contract trucks and temporary staff at the top of it. How do you respond?