Core day 25 of 60 · Week 5 · Part I · evaluation depth

Time-Aware Validation

A gradient-boosting model beats seasonal naive by 40% in 5-fold cross-validation and a colleague wants to ship it. What did each fold let the model see, what must it never see, and which evaluation tells you the error to expect 14 days ahead in production?

By the end: You draw a rolling-origin backtest for a 14-day horizon, mark for each fold what the model sees and what it must never see, and report error by horizon and across origins as a ratio to the seasonal-naive baseline.

At a glance

A forecast is validated by replaying deployment, cutoff by cutoff, at the decision horizon, against the baseline on the same origins.

Simulate deployment
Each fold sees only data dated before its cutoff and is scored after it.
No future in training
Random k-fold trains on days after the test day, so it interpolates.
Gap
Leave out at least the span of features or labels that reach across the cutoff.
By horizon and origin
Error grows with the horizon; report share of origins won and the worst ones.
Ratio to baseline
Model MAE over seasonal-naive MAE on identical origins decides; MASE only scales.

Recall · 5 min

Without looking back: on Day 24, which two properties must a lag feature for a 14-day-ahead model have, and what does a rolling window that ends at t do to the validation score?

Open due reviews

Concept · 10 min

A validation scheme is a simulation of deployment. In production you stand at a cutoff, know everything dated before it, forecast the days after it, and are scored later. A valid fold reproduces that: the model sees only the past, never anything after the cutoff, and is scored at the horizon you will use. Random k-fold breaks the second rule.

Random k-fold lets the model see the future; each time-aware scheme draws a different line

Predict first

With random k-fold selected, how many training rows typically lie after the first test row of a fold?

Text description of the figure

A selector for the scheme (random k-fold, time-series split, sliding window, expanding window, rolling origin), sliders for horizon, gap and training window, and a fold stepper. A strip of 48 time points shows each fold with cells for train, gap, test, never-seen future and unused older points; all folds are stacked below with the selected one outlined. A sentence lists the train, gap and test positions, and a readout counts training rows that lie after the first test row.

Three rules every fold must follow Overview

Sees only the past
Training data dated before the cutoff, minus any gap.
Never sees the future
Nothing dated after the cutoff, including the days next to a test day.
Scored at your horizon
The horizon the decision uses, against the baseline on the same origins.

Four schemes, drawn fold by fold Illustration

Test points are scattered, so most training points are later than some test point. Training points after the first test point: 92 of 108.

The scikit-learn layout: test blocks at the end, training grows, a gap before each test block. Training points after the first test point: 0 of 96.

All history up to the gap; more folds than the split, same rule. Training points after the first test point: 0 of 100.

Old points leave the training set: useful after a break, fewer rows per fit. Training points after the first test point: 0 of 60.

train (before the test)
train dated after the fold's first test point
gap
test
not used

Folds generated by the course's fold builder for 36 points, horizon 4, gap 2, at least 12 training points; the shuffle is seeded.

Folds built by the course's fold builder on 36 time-ordered points. Red cells are training points dated after the fold's first test point; only shuffling produces them.

Why a shuffled fold flatters a forecaster Illustration

The test day (ringed) sits between known days on both sides when rows are shuffled. At a real cutoff 14 days earlier, everything to its left after the cutoff and everything to its right is unknown.

Averaging the neighbours on both sides gives MAE 3.0, 30% below the honest 14-day guess (4.3). That gap is what a shuffled fold can hand a model with a time index or day-of-year feature, without any forecasting skill.

Synthetic daily demand (seeded, three years): a level that drifts as a random walk in logs, a weekly shape and 3% noise. No model is fitted; the three guesses use the raw values that each situation makes available.

With the days on both sides of a test day available, even a plain average beats an honest 14-day-ahead guess. A model with a time index finds that shortcut by itself.

A rolling-origin backtest replays deployment, one origin at a time Loop

Move the origin c_ke.g. one week latercut the data ↓
Cut at c_k - gapnothing dated later entersrebuild ↓
Refit everythingfeatures, scalers, tuning, vintagesforecast ↓
Forecast h stepsthe decision's horizonscore ↓
Score by horizonloss vs actuals and the baselinenext origin ↓
↺ back to 1 · Move the origin c_k

The estimate is honest only if every step production repeats is repeated inside each origin, and if the evaluation window resembles the future: a full year when demand is seasonal (winter peaks in energy load, holiday weeks in retail). Pool errors by horizon, never across horizons.

Error grows with the horizon (hands-on lab, rolling origin) Chart

Lag-based direct model
Seasonal naive at that horizon

Ratio to seasonal naive: 0.71 at 1 day, 0.88 at 7, 0.93 at 14, 0.91 at 28. A pooled number would mix the easy horizons with the one the decision uses.

Same model, two schemes: ratio of model MAE to seasonal-naive MAE (today's run) Chart

Random 5-foldwins 68% of 364 forecasts: interpolation0.6
Rolling origin, h = 14wins 46% of 52: the production number1.03

Below 1 means the model beats the baseline. Calendar fields and a time index let trees interpolate between neighbours; a real cutoff removes the neighbours.

Schemes: expanding window, sliding window and rolling origin keep time order; random k-fold does not.
Gap (embargo): leave out at least the span of any feature or label that reaches across time.
By horizon and by origin: score at the decision horizon; report the share of origins won and the worst ones.
Skill: the ratio of model MAE to seasonal-naive MAE on the same origins; MASE is a scale, not a verdict.

Assumptions

The evaluation window resembles deployment, so it covers the seasons and regimes the model will face, with at least a full year when the season is annual.
Labels are final at each cutoff; if actuals are revised later, backtests must use the values that existed then.
There are enough origins to see variation; a handful gives a number with a wide, unstated error bar.

Application and workplace decision · 10 min

The code simulates three years of daily demand with a weekly shape and a drifting level, and scores a LightGBM model on calendar fields and a time index against seasonal naive at horizon 14 on the last year, twice. Before you run it, predict the ratio of model MAE to baseline MAE under each scheme.

Scheme 1: random 5-fold, scored on the days of the last year.
Scheme 2: a rolling origin that refits weekly on labels up to the cutoff and forecasts day c+14.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
import pandas as pd
from lightgbm import LGBMRegressor
from sklearn.model_selection import KFold

rng = np.random.default_rng(5)
n, h, first = 3 * 364, 14, 728                  # three years of daily demand; forecasts made 14 days ahead
t = np.arange(n)
level = 100 * np.exp(np.cumsum(rng.normal(0.0004, 0.012, n)))   # slowly drifting level (random walk in logs)
week = np.array([1.00, 0.95, 0.93, 0.97, 1.08, 1.12, 1.05])
y = level * week[t % 7] * (1 + rng.normal(0, 0.03, n))
X = pd.DataFrame({"t": t, "dow": t % 7, "doy": t % 364})        # calendar + time index: all known in advance
naive = np.r_[np.full(h, np.nan), y[:-h]]       # seasonal naive at horizon 14: the same weekday two weeks ago
scale = np.abs(y[7:first] - y[:first - 7]).mean()   # in-sample seasonal-naive MAE: the MASE denominator
model = lambda: LGBMRegressor(n_estimators=200, learning_rate=0.05, num_leaves=15, min_child_samples=5,
                              n_jobs=1, verbose=-1)

k_err, k_base = [], []                          # 1) random 5-fold, scored on the days of the last year
for tr, te in KFold(5, shuffle=True, random_state=0).split(X):
    te = te[te >= first]
    k_err += list(np.abs(y[te] - model().fit(X.iloc[tr], y[tr]).predict(X.iloc[te])))
    k_base += list(np.abs(y[te] - naive[te]))

r_err, r_base = [], []                          # 2) rolling origin: fit on labels up to c, forecast day c + 14
for c in range(first - h, n - h, 7):
    p = model().fit(X.iloc[: c + 1], y[: c + 1]).predict(X.iloc[[c + h]])[0]
    r_err.append(abs(y[c + h] - p))
    r_base.append(abs(y[c + h] - naive[c + h]))

for name, e, b in [("random 5-fold", k_err, k_base), ("rolling origin", r_err, r_base)]:
    e, b = np.array(e), np.array(b)
    print(f"{name:15s} MAE model {e.mean():5.1f} vs seasonal naive {b.mean():5.1f} | ratio {e.mean() / b.mean():.2f} | "
          f"MASE model {e.mean() / scale:.2f}, naive {b.mean() / scale:.2f} | model wins {np.mean(e < b):.0%} of {len(e)}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 22 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

Rolling origin as repeated out-of-sample estimation5 minThe estimand a backtest targets, three origins computed by hand, and why 52 correlated origins carry the information of far fewer.

A rolling-origin backtest is an estimator of the deployed error at horizon h Formula

\widehat{R}(h) = \frac{1}{K}\sum_{k=1}^{K} L\big(y_{c_k+h},\ \hat y_{c_k+h \mid c_k}\big) \quad \text{estimates} \quad R(h) = \mathbb{E}\big[L(y_{c+h},\ \hat y_{c+h \mid c})\big]

$c_k$: the k-th origin (cutoff); everything used to build the forecast is dated at or before c_k, minus the gap
$\hat y_{c_k+h \mid c_k}$: the forecast the whole procedure - features, refit, tuning - would have issued at c_k for h steps later
$L$: the loss the decision cares about: absolute error, squared error, pinball (Day 16)
$R(h)$: the estimand: expected loss of the procedure at horizon h over cutoffs like the ones you will face in production
$K$: number of origins; more of them shrink the variance of the estimate, never its bias

The object being estimated is the error of a procedure, not of one fitted model: refit at each origin is part of it. A random fold estimates a different quantity - the error of filling a hole inside known history.

Worked by hand: naive forecasts of weekly call volume from three origins Table

Origin c	Last value y_c	h = 1: actual, abs error	h = 2: actual, abs error
Weeks 1-8 (hundreds of calls)	20, 22, 21, 25, 24, 26, 27, 25	-	-
c = 4	25	y_5 = 24, error 1	y_6 = 26, error 1
c = 5	24	y_6 = 26, error 2	y_7 = 27, error 3
c = 6	26	y_7 = 27, error 1	y_8 = 25, error 1
R_hat(h) = mean over origins	-	(1 + 2 + 1) / 3 = 1.33	(1 + 3 + 1) / 3 = 1.67

Illustrative contact-centre volumes, naive forecast y_hat = y_c at every horizon. The error is pooled by horizon, never across horizons, and week 7 appears as a target twice (c = 5 at h = 2, c = 6 at h = 1): origins share data, so their errors are correlated.

How many independent origins 52 weekly origins are worth Formula

\operatorname{Var}\big(\widehat{R}\big) \approx \frac{s^{2}}{K}\Big(1 + 2\sum_{j\ge 1}\rho_j\Big), \qquad \rho_j = \rho^{j}:\ K_{\text{eff}} \approx K\,\frac{1-\rho}{1+\rho}

$s^{2}$: variance of one origin's loss
$\rho_j$: correlation of the losses of origins j apart: overlapping target windows and shared regimes make it positive
$K_{\text{eff}}$: the same AR(1) effective-sample-size algebra as Day 22, applied to the sequence of per-origin errors

Worked: 52 weekly origins at h = 14 with an illustrative lag-1 error correlation of 0.5 give K_eff = 52 x 0.5 / 1.5 = 17.3, so a naive standard error is sqrt(52 / 17.3) = 1.7 times too small. Quote the spread, the share won and an interval from K_eff; daily instead of weekly origins add little when h = 14.

Ratio, skill and MASE, worked by hand4 minThe MASE formula, eight quarters of sales scored by hand, and which number answers "does it beat the baseline".

MASE: test error over the in-sample seasonal-naive error Formula

\text{MASE} = \frac{\frac{1}{H}\sum_{j=1}^{H} |y_{T+j} - \hat y_{T+j}|}{\frac{1}{T-m}\sum_{t=m+1}^{T} |y_t - y_{t-m}|}

$\text{numerator}$: the model's MAE on the test block
$\text{denominator}$: in-sample MAE of a one-season-back forecast on the training data: a scale, fixed before the test
$m$: the seasonal period (1 gives the non-seasonal naive scale)

Scale-free, so it averages across series of different sizes (Hyndman and Koehler, 2006): over thousands of SKUs or hundreds of hospital wards a big series no longer dominates the mean. It is not the ratio to the baseline on the same test block, which is the shipping question.

Worked by hand: MASE and the baseline ratio on quarterly sales (illustrative) Table

Step	Numbers	Result
Training, 8 quarters	10, 14, 12, 18 \| 11, 16, 13, 20	m = 4
Scale: \|y_t - y_(t-4)\|, t = 5..8	\|11-10\|, \|16-14\|, \|13-12\|, \|20-18\| = 1, 2, 1, 2	mean 1.5
Test actuals, next 4 quarters	12, 17, 14, 21	-
Model forecast, abs errors	13, 16, 14, 20 -> 1, 1, 0, 1	MAE 0.75, MASE 0.75 / 1.5 = 0.50
Seasonal naive (last year), abs errors	11, 16, 13, 20 -> 1, 1, 1, 1	MAE 1.00, MASE 0.67
Ratio to the baseline, same quarters	0.75 / 1.00	0.75: the model is 25% better

A retailer's product-line sales in thousands of units. Both forecasts have MASE below 1 because the test year happened to be calm; only the ratio on the same quarters compares them. Recomputed in Python.

Skill relative to the baseline Formula

\text{ratio} = \frac{\text{MAE}_{\text{model}}}{\text{MAE}_{\text{seasonal naive}}}\Big|_{\text{same origins, same horizon}}, \qquad \text{skill} = 1 - \text{ratio}

$\text{ratio} < 1$: the model beats the baseline on these origins
$\text{MASE}$: scales by the in-sample one-step seasonal-naive error: unit-free, but a different statement

At long horizons both model and baseline exceed MASE 1, which is why MASE alone cannot answer "should we ship".

The schemes compared, and the gap3 minWhat each scheme trains on, whether it sees the future, when to use it, and what a gap does to a label that spans seven days.

Four schemes, what each one sees Comparison

Criterion	Random k-fold	Expanding window	Sliding window	Rolling origin
Training data	A random 80% of rows	Everything before the test block	Only the last W points	Everything before each origin
Future in training?	Yes	No	No	No
Training size	Constant	Grows	Constant	Grows (or slides)
Use when	Exchangeable rows; pure AR models with white-noise errors	Stable process; little data	Drift or regime changes	Error by horizon; the forecasting default

scikit-learn's TimeSeriesSplit is the expanding layout with test blocks at the end. A sliding window helps after a break (Day 21) or under drift; the price is a smaller training set and a season that may be poorly covered.

A label spanning the next 7 days: with and without a gap (hands-on lab) Chart

TimeSeriesSplit, gap = 0last training labels overlap the test block6.61 MAE
TimeSeriesSplit, gap = 7the number production would see8.92 MAE

The lower score is the leaky one: without a gap the last training labels are computed from test-period days. TimeSeriesSplit(gap=...) drops that many samples from the end of each training set; size it to the longest span of a feature window or label. Next-week call volume, next-7-day claims and a 30-day churn flag all span time this way.

Stability across origins3 minA pooled ratio can hide losing quarters, and one series can hide a lucky draw.

What a rolling-origin backtest really returns Illustration

Mean error grows from 4.0 at step 1 to 6.2 at step 12, and single origins scatter far around it. Ratio to seasonal naive by step: 0.63 at 1, 1.05 at 6, 0.97 at 12.

Pooled ratio 0.95: the model wins 16 of 25 origins, but its worst three (origins 100, 102, 60) have 3.4, 3.3, 2.5 times the baseline error. A pooled number hides that tail.

Synthetic monthly series (seeded, 10 years): trend, a 12-month season, autocorrelated noise and a level shift of +9 at month 88. Both methods refitted at every second origin from month 60, forecasting 12 months; the AR model regresses each month on the previous 13 by least squares.

Errors by horizon for every origin (left) and each origin's ratio to seasonal naive (right). The mean hides both the spread and the tail.

Ratio to seasonal naive at h = 14 by quarter of the test year (hands-on lab) Chart

Quarter 10.76
Quarter 2loses1.08
Quarter 3loses1.09
Quarter 40.87

The pooled ratio is 0.93, yet the model loses half the year. Report the slices and the worst origins, not only the mean.

Is the flip a lucky draw? Ratio to seasonal naive on six simulated series (hands-on lab) Grid

Scheme →
Series seed ↓	Random k-fold	Rolling origin
1	0.63	1.42
2	0.67	1.10
3	0.62	0.96
4	0.65	0.93
5	0.61	1.09
6	0.60	1.14

Below 1 means the model beats the baseline. Random k-fold promises 33-40% gains every time; the rolling origin shows a loss on four of six series. Count every variant tried: the best of many backtests is optimistic.

A checklist before quoting a backtest2 minFour questions, in order, that decide whether a number can be quoted.

Before anyone quotes a backtest number Diagram

Training rows after the test?
- no: to Gap >= window span?
yes
Invalid for forecastingrebuild with time order
Gap >= window span?
- yes: to At the decision horizon?
no
Add the gaplongest feature or label span
At the decision horizon?
- yes: to Ratio < 1 across origins?
no
Rescore by horizonpooled numbers mix horizons
Ratio < 1 across origins?
- yes: to Candidate to ship
no
Keep the baselineor build features that forecast
Candidate to shipwith share won and worst origins

Each fix sends you back to rerun the backtest; only a number that passes all four questions is quoted.

Backtests in scikit-learn and statsforecastadvanced2 minThe default fold layout of both libraries, the arguments that make it match deployment, and what refits cost.

Backtest knobs in the two libraries used in Part I Table

Call	Defaults	What to set
sklearn TimeSeriesSplit	n_splits=5, test_size=n // (n_splits + 1), gap=0, max_train_size=None	test_size = horizon; gap >= longest feature or label span; max_train_size for a sliding window
statsforecast cross_validation	n_windows=1, step_size=1, refit=True	h = decision horizon; n_windows to cover a full season; refit=False only to save time
sklearn Pipeline in cross_val_score	cv=TimeSeriesSplit(...)	Fits scalers, encoders and feature windows inside each origin

Read from scikit-learn 1.9.1 and statsforecast 2.0.1. TimeSeriesSplit's test blocks sit at the end of the data: for many origins across a year, loop over origins or use n_windows. Refits cost K x fit time: cheap for ETS, real money for a global GBM over 10,000 SKUs, hence refit=False or a refit every few origins.

Reporting, and the ways a valid backtest still flatters3 minThe rows a trustworthy report contains, and four sources of optimism that survive a correct fold layout.

A backtest report a reviewer can trust Table

Row	Why it is there
MAE by horizon, model and baseline	The decision uses one horizon; pooled numbers mix easy and hard steps
Ratio to the baseline, same origins	The shipping question, at the decision horizon
Share of origins won	Stability: a 0.95 ratio from 50% wins is a coin flip
Three worst origins with dates	Where and when the model fails; often a break or a holiday
Bias (mean error)	Accuracy can hide a systematic over-forecast
Variants tried before this one	Every extra try makes the best one more optimistic
One final untouched period	Scored once, after every choice: the number to quote

How a valid backtest still ends up optimistic Overview

Many variants
The best of 50 feature sets on the same origins is partly luck.
Tuning on the test origins
Hyperparameters picked where they are scored; tune on an earlier window.
Labels as known later
Restated actuals in training; use the vintage that existed at each cutoff.
Correlated origins
Weekly origins overlap in time; 52 of them are not 52 independent trials.

Memory anchor

Draw every fold, mark what lies after the cutoff, and report the ratio to the baseline at the horizon you will use.

Useful for

Producing the number that production will show, for any forecasting model, with a stated horizon, gap and baseline.
Comparing model families (ETS, ARIMA, boosting, deep) on identical origins, and choosing between expanding and sliding windows.
Reviewing a colleague's evaluation by drawing the folds and marking what the model saw.
Any decision fed by a forecast at a fixed lead time - day-ahead energy load, two-week retail replenishment, next-month hospital bed demand, quarterly churn in telecom.

Not suitable for

Exchangeable rows with no temporal dependence (many cross-sectional problems), where grouped or stratified k-fold is the right tool.
Tiny histories with only a few possible origins - report that the backtest cannot tell much, and prefer simple models.

Traps

One pooled number across horizons and origins

A single MAE or ratio averages short and long horizons and hides origins where the model fails badly. The model "wins on average" but loses on the horizon and the periods that matter - the winter peak of a load forecast, the holiday weeks of a retailer.

How to avoid it

Report the ratio to the baseline by horizon bucket and by period of the year, with the share of origins won and the worst origin.

No gap when features or labels span time

Rows at the end of training carry rolling-window features or labels that overlap the test block. The model has effectively seen part of the test period, and scores improve without any forecasting skill.

How to avoid it

Set the gap to at least the longest feature window or label span, and rebuild features inside each origin.

Senior reasoning

A senior designs the backtest from the deployment: cutoff, horizon, labels as they were known then, a window that covers the seasons that matter, and a gap. They report the model against the baseline at the decision horizon on the same origins and say how many origins that rests on. They treat every extra variant tried as a cost to the credibility of the best result, and they keep a final untouched period for the claim they will defend.

Interview question

"Your boosting model beats the baseline in cross-validation but not in production. What went wrong?" Structure the answer.

Outline your answer first.

Recall cards scheduled after completion

For each of random k-fold, expanding, sliding and rolling origin, say what the model sees and what it must never see.
What do you report to say a model beats seasonal naive, and why is MASE not enough?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Forecasting: Principles and Practice (3rd edition) — Rob J Hyndman, George Athanasopoulos, 2021core
Time-series cross-validation on a rolling forecasting origin, with forecast error rising with the horizon in the worked example; the test set should be at least as long as the longest horizon; MASE definition. (Sections 5.8 and 5.10)
sklearn.model_selection.TimeSeriesSplit (scikit-learn API reference) — scikit-learn developerscore
The gap, test_size and max_train_size arguments, which remove samples before each test block and cap the training window.
A note on the validity of cross-validation for evaluating autoregressive time series prediction — Christoph Bergmeir, Rob J. Hyndman, Bonsoo Koo, 2018optional
Standard K-fold cross-validation is valid for purely autoregressive models when residuals are uncorrelated, a nuance to "never shuffle".
Evaluating time series forecasting models: an empirical study on performance estimation methods — Vitor Cerqueira, Luis Torgo, Igor Mozetič, 2020optional
In an empirical study of 174 real series plus synthetic series, blocked cross-validation suited stationary series while repeated out-of-sample tests estimated performance best for non-stationary ones.
Out-of-sample tests of forecasting accuracy: an analysis and review — Leonard J. Tashman, 2000optional
Review of out-of-sample forecast evaluation designs - rolling origins, fixed and rolling windows, multiple horizons.
Another look at measures of forecast accuracy — Rob J. Hyndman, Anne B. Koehler, 2006optional
MASE scales errors by the in-sample naive or seasonal-naive mean absolute error.

Optional sections

Deep dive · 10 min

Why leakage works in random folds. If the level has memory (autocorrelation), the labels of the days around a test day carry most of its value. A model with a time index, day of year or an entity-time interaction can encode those neighbours; with lag features of at least h it cannot, which is why the same random fold is less misleading for a correctly built direct model, though still not a forecast evaluation.

Expanding vs sliding. An expanding window uses all history and is stable when the process is stable. A sliding window drops old regimes and fits faster, but may lose rare seasonal events (annual holidays need enough years). Backtest both and two or three window lengths on recent origins; do not choose by habit.

Fixed-origin vs rolling-origin. One holdout of the last 12 months is one draw. Rolling origins reuse the same data to give many draws that are correlated, so they reduce variance but do not give independent samples; report the spread across origins and avoid reading them as independent trials.

Embargo in forecasting. The gap is a feature-window issue and a label-maturation issue. For a model with horizon h trained at cutoff c, labels at t > c are unknown; with features that look back w days, rows whose windows overlap the test block should also be dropped. Late-arriving or revised actuals need vintages: evaluate with the label as it was known at c. Metrics. Per-origin MAE and the ratio to seasonal naive; MASE for comparing across series; bias (mean error) separately from accuracy; for intervals, coverage by horizon (Day 26). For hierarchies evaluate at each level (Day 26). With many series, report the distribution of ratios across series, not only their mean.

Counting trials. Every feature set, window and hyperparameter tried on the backtest adds a chance that the best one is luck. Keep a log, hold out a final period used once, and prefer settings that win across many origins to a sharp optimum.

Hands-on lab: Error by horizon, the embargo, and a seed check of the flip · 25 min

Run the code and answer: (1) At which horizon is the lag-based model's ratio to seasonal naive closest to 1, and in which quarter(s) of the test year does it lose at h = 14? (2) In part 2 the label is the mean of the next seven days. Why does gap = 0 give the lower MAE, and which number would production see? (3) In part 3, in how many of six series does the rolling-origin ratio exceed 1 while the random k-fold ratio is below 0.7? What does that say about the colleague's claim? (Seed 5 is the application series with 100 trees and cutoffs every 14 days, so its ratio differs slightly from the application's 1.03.) (4) Change the level volatility 0.012 to 0.004 in make and describe how the flip changes, and why.

import numpy as np
import pandas as pd
from lightgbm import LGBMRegressor
from sklearn.model_selection import KFold, TimeSeriesSplit

n, first = 3 * 364, 728
t = np.arange(n)
week = np.array([1.00, 0.95, 0.93, 0.97, 1.08, 1.12, 1.05])

def make(seed):
    rng = np.random.default_rng(seed)
    level = 100 * np.exp(np.cumsum(rng.normal(0.0004, 0.012, n)))
    return level * week[t % 7] * (1 + rng.normal(0, 0.03, n))

def gbm():
    return LGBMRegressor(n_estimators=100, learning_rate=0.05, num_leaves=15, min_child_samples=5,
                         n_jobs=1, verbose=-1)

# 1) error by horizon: direct model with lags >= h on level-free ratios vs the seasonal naive of that horizon
y = make(5)
s = pd.Series(y)
print("horizon  MAE model  MAE seasonal naive  ratio   ratio by quarter of the test year")
for h in (1, 7, 14, 28):
    lvl = s.shift(h).rolling(7).mean()                       # level known at the cutoff
    X = pd.DataFrame({"r_h": s.shift(h) / lvl, "r_h7": s.shift(h + 7) / lvl, "dow": t % 7})
    naive = s.shift(7 * int(np.ceil(h / 7))).values          # most recent same weekday known at the cutoff
    err, base = [], []
    for c in range(first - h, n - h, 7):                     # cutoff c, target c + h, labels up to c only
        ok = np.arange(28 + h, c + 1)
        m = gbm().fit(X.iloc[ok], (s / lvl).values[ok])
        err.append(abs(y[c + h] - lvl[c + h] * m.predict(X.iloc[[c + h]])[0]))
        base.append(abs(y[c + h] - naive[c + h]))
    e, b = np.array(err), np.array(base)
    q = [e[i].mean() / b[i].mean() for i in np.array_split(np.arange(len(e)), 4)]
    print(f"{h:5d}   {e.mean():9.1f}  {b.mean():18.1f}  {e.mean() / b.mean():5.2f}   {np.round(q, 2)}")

# 2) embargo: the label is the mean of the NEXT 7 days, so the last training labels overlap the test block
label = s.rolling(7).mean().shift(-7).values                 # mean of y[t+1..t+7]
X = pd.DataFrame({"t": t, "dow": t % 7, "lag_1": s.shift(1), "lag_7": s.shift(7)})
ok = ~np.isnan(label) & X.notna().all(axis=1).values
Xo, lo = X[ok], label[ok]
for gap in (0, 7):
    maes = []
    for tr, te in TimeSeriesSplit(n_splits=8, test_size=21, gap=gap).split(Xo):
        maes.append(np.abs(lo[te] - gbm().fit(Xo.iloc[tr], lo[tr]).predict(Xo.iloc[te])).mean())
    print(f"TimeSeriesSplit gap = {gap}: MAE {np.mean(maes):.2f}")

# 3) is the flip a lucky draw? random 5-fold vs rolling origin (h = 14, calendar + time index) on six series
for seed in range(1, 7):
    yy = make(seed)
    X = pd.DataFrame({"t": t, "dow": t % 7, "doy": t % 364})
    nv = np.r_[np.full(14, np.nan), yy[:-14]]
    ke, kb = [], []
    for tr, te in KFold(5, shuffle=True, random_state=0).split(X):
        te = te[te >= first]
        ke += list(np.abs(yy[te] - gbm().fit(X.iloc[tr], yy[tr]).predict(X.iloc[te])))
        kb += list(np.abs(yy[te] - nv[te]))
    re, rb = [], []
    for c in range(first - 14, n - 14, 14):
        re.append(abs(yy[c + 14] - gbm().fit(X.iloc[: c + 1], yy[: c + 1]).predict(X.iloc[[c + 14]])[0]))
        rb.append(abs(yy[c + 14] - nv[c + 14]))
    print(f"seed {seed}: random k-fold ratio {np.mean(ke) / np.mean(kb):.2f}, "
          f"rolling-origin ratio {np.mean(re) / np.mean(rb):.2f}")

Interview practice · 10 min

A teammate at an energy retailer shows a rolling-origin backtest of daily load with one cutoff at the end of last year and says "it is time-aware, so it is valid". What is missing, and what would you ask for?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-18 · depends on scikit-learn, lightgbm, numpy, pandas