Core day 22 of 60 · Week 5 · Part I · application depth

Autocorrelation and Transformations

A monthly passenger series trends upward and its seasonal swings grow with the level. Before fitting anything: which transformation and differencing steps make it analysable, how do you read its autocorrelation bars, and what can a clean diagnostic still not tell you about the model?

By the end: Given a series you choose a log or Box-Cox step and the number of ordinary and seasonal differences, read ACF bars against the white-noise band, and explain why a clean ACF or Ljung-Box result does not prove the model is suitable.

At a glance

Tame the swings, difference as little as needed, then read the autocorrelation bars against the noise band and test them jointly.

ACF and PACF
Total echo at lag k versus the direct link once the lags in between are removed.
The band
Plus or minus 1.96 over root T; one stray bar in 24 is what noise does.
Transform
Log or Box-Cox when swings grow with the level; back-transforming gives the median.
Difference
Seasonal then ordinary, as few as needed; lag 1 near -0.5 with a larger spread means too many.
Clean is not good
White-noise residuals are necessary; only a backtest shows the model beats the baseline.

Recall · 5 min

Without looking back: why does a rolling mean that drifts not tell you whether the cause is trend or a break, and which stationarity test has "the series is stationary" as its null hypothesis?

Open due reviews

Concept · 10 min

The autocorrelation function is the series' echo: its correlation with itself k steps back. The partial autocorrelation is the direct voice: the lag-k link after removing the lags in between. Memory shows at small lags, a season at multiples of its period. Read them only after a transformation tames growing swings and differencing removes drift; on a trending series every bar is large.

Every process leaves a different ACF and PACF signature - and short noisy series blur all of them

Predict first

You select AR(1) with phi = 0.7 and length 200. What does the PACF show?

Text description of the figure

A selector for the simulated process (white noise, AR(1), MA(1), seasonal, random walk), sliders for the AR or MA coefficient and the series length, and a button for a new random draw. Three panels show the simulated series and the ACF and PACF as stems for lags 1 to 24 with a dashed band at plus and minus 1.96 over the square root of the length. A readout gives the number of ACF lags outside the band, the Ljung-Box statistic for 10 lags and the band half-width.

Five tools for today Overview

ACF
Correlation of the series with itself k steps earlier: the total echo.
PACF
The lag-k link after removing the lags in between: the direct voice.
White-noise band
Plus or minus 1.96 over root T: about 1 bar in 20 falls outside by chance.
Transform
Log or Box-Cox tames swings that grow with the level.
Difference
Removes level wander (d) and a fixed seasonal pattern (D).

Six processes, six signatures Illustration

Nothing systematic: stray bars only. Band ±0.13; ACF outside at 1 of 24 lags; PACF outside at lag 16.

ACF decays geometrically; PACF cuts off after lag 1. Band ±0.13; ACF outside at 8 of 24 lags; PACF outside at lag 1.

ACF cuts off after lag 1; PACF alternates and decays. Band ±0.13; ACF outside at 2 of 24 lags; PACF outside at lags 1, 2, 3, 7.

Spikes at 12 and 24 in the ACF; the large PACF bar is at 12. Band ±0.13; ACF outside at 4 of 24 lags; PACF outside at lags 2, 12, 13.

ACF starts near 1 and barely decays: difference first. Band ±0.13; ACF outside at 24 of 24 lags; PACF outside at lag 1.

Same process, short series: wider band, noisier bars. Band ±0.31; ACF outside at 3 of 24 lags; PACF outside at lag 1.

Simulated processes (seeded, 240 points unless stated, standard normal shocks). Bars are sample ACF and PACF (Durbin-Levinson) from the course's time-series code; vertical scale -1 to 1; the shaded band is ±1.96/√T. Thick bars lie outside it.

ACF (top) and PACF (bottom) of simulated series, lags 1 to 24. Read the shape, not single bars; the last panel shows how a short series blurs the same process.

Signatures to recognize Table

Process	ACF	PACF
White noise	All bars inside the band	All bars inside the band
AR(p)	Decays geometrically	Cuts off after lag p
MA(q)	Cuts off after lag q	Decays
Season with period m	Spikes at m, 2m, ...	Spike at m
Random walk	Starts near 1 and decays very slowly	One large bar at lag 1

Signatures are clear on long simulated series and blur on short noisy ones; the widget shows both.

Prepare a series in a fixed order, one question at a time Diagram

Raw seriesplot it first
Swings grow with level?
- yes: to Log or Box-Cox
no
Seasonal pattern left?
- yes: to Seasonal difference, D = 1
no
Level still wanders?
- yes: to First difference, d = 1
no
Stop: read ACF and sdsmallest spread wins
Log or Box-Coxstrictly positive data
Seasonal difference, D = 1y(t) - y(t-m)
First difference, d = 1y(t) - y(t-1)

After each action on the right, continue with the next question down. Seasonal differencing comes first because it sometimes removes the level drift as well. Stop at the step where the standard deviation stops falling.

The ladder: one step at a time Illustration

lag 1 +0.94, lag 12 +0.78, sd 55.5; swings grow with the level.

lag 1 +0.94, lag 12 +0.79, sd 0.326.

lag 1 +0.43, lag 12 -0.35, sd 0.039; smallest spread: stop here.

lag 1 -0.39, lag 12 -0.39, sd 0.042.

lag 1 -0.61, lag 12 -0.38, sd 0.071; spread grows: one difference too many.

Synthetic monthly passengers (12 years, seeded): drifting log level, a season that scales with the level and evolves slowly, small noise. Each panel applies one more step and has its own vertical scale; sd is the standard deviation of the transformed series (the raw one is in passengers). On this draw the seasonal difference alone gives the smallest spread; on the application series it takes D = 1 and d = 1, so read the rule per series.

Each panel adds one transformation to the same series and recomputes the ACF and the spread. Stop at the rung where the spread is smallest.

Back-transforming a log forecast gives the median; the mean needs a bias step Bridge

exp(mean of log) = median100.4 passengers
Bias adjustment, exp(s^2/2)half the log-scale variance, 0.5^2 / 2+13.3 passengers
Mean, exp(m + s^2/2)113.7 passengers

A log-scale forecast distribution with standard deviation 0.5, 100,000 draws (today's run); the simulated mean of exp is the same 113.7. Medians do not add across routes, stores or meters; means do.

A clean residual ACF is one filter of three Stack

Residual ACF and Ljung-BoxSees linear pattern left in the past sample. Blind to skill at the horizon and to anything after the fit.
Rolling-origin backtestSees whether the model beats seasonal naive at the plan's horizon (Days 23 and 25).
Monitoring after launchSees breaks after the fitting window, such as a run of same-sign errors (Day 21).

Necessary, not sufficient: the lab's AR(1) passes the first filter and still fails after a break (Ljung-Box chapter).

Band: White noise keeps about 95% of bars inside $\pm 1.96/\sqrt{T}$ ; one stray bar in 24 is expected.
Joint test: the Ljung-Box test on residuals, with the model's parameters removed from the degrees of freedom.
Transform: log or Box-Cox when swings scale with the level; mind back-transformation bias.
Difference: seasonal (D) and ordinary (d), as few as needed. Decompose with STL.

Assumptions

The ACF bars are read after the series is made roughly stationary; on a trending series every bar is large and uninformative.
The band $\pm 1.96/\sqrt{T}$ is valid for white noise; for a fitted model's residuals and for short series it is only a rough guide.
The seasonal period is known and fixed, and the series is strictly positive when a log or Box-Cox step is used.

Application and workplace decision · 10 min

The code builds 12 years of synthetic monthly passengers (multiplicative season, drifting level) and applies one step at a time. Before running, predict which step first removes the positive lag-12 correlation and which one raises the standard deviation.

Each step prints the lag-1 and lag-12 ACF and the standard deviation.
Then it fits the classic airline model and tests its residuals with Ljung-Box.
Finally it measures the back-transformation gap for a noisy log-scale forecast.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
from statsmodels.stats.diagnostic import acorr_ljungbox
from statsmodels.tsa.stattools import acf
from statsmodels.tsa.statespace.sarimax import SARIMAX

rng = np.random.default_rng(22)
n, m = 144, 12                                       # 12 years of synthetic monthly passengers
level = np.log(100) + np.cumsum(0.006 + rng.normal(0, 0.012, n))                  # drifting level
pattern = 0.25 * np.sin(2 * np.pi * np.arange(m) / m)
season = (pattern + np.cumsum(rng.normal(0, 0.01, (n // m, m)), axis=0)).ravel()  # evolving season
log_y = level + season + rng.normal(0, 0.02, n)
y = np.exp(log_y)                                    # swings grow with the level (multiplicative)

def diff(x, lag=1):
    return x[lag:] - x[:-lag]

steps = {"raw": y, "log": log_y, "log, d=1": diff(log_y), "log, D=1": diff(log_y, 12),
         "log, D=1, d=1": diff(diff(log_y, 12)), "log, D=1, d=2": diff(diff(diff(log_y, 12)))}
print(f"white-noise band for n = {n}: +-{1.96 / np.sqrt(n):.2f}")
for name, x in steps.items():
    r = acf(x, nlags=24)
    print(f"{name:15s} lag1 {r[1]:+.2f}  lag12 {r[12]:+.2f}  sd {x.std():.3f}")

fit = SARIMAX(log_y, order=(0, 1, 1), seasonal_order=(0, 1, 1, m)).fit(disp=False)
res = fit.resid[m + 1:]                              # drop the start-up residuals created by differencing
p = acorr_ljungbox(res, lags=[24], model_df=2).loc[24, "lb_pvalue"]
print(f"SARIMA(0,1,1)(0,1,1)12 residuals: Ljung-Box(24, model_df=2) p = {p:.2f}")

e = rng.normal(np.log(100), 0.5, 100_000)            # a log-scale forecast distribution with sd 0.5
print(f"exp(mean log) = {np.exp(e.mean()):.1f}; mean of exp = {np.exp(e).mean():.1f}; "
      f"exp(m + s^2/2) = {np.exp(e.mean() + e.var() / 2):.1f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 24 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The ACF as an estimator, and the AR(1) echo derived4 minThe sample ACF formula, six points pushed through it by hand, why an AR(1)'s correlations decay as phi to the k, and the band that widens.

The sample ACF is an estimator Formula

r_k = \frac{\sum_{t=k+1}^{T}(y_t-\bar y)(y_{t-k}-\bar y)}{\sum_{t=1}^{T}(y_t-\bar y)^{2}}

$r_k$: the estimate, from one realization, of the process property rho_k
$\bar y$: the full-sample mean
$T$: the denominator always sums all T squared deviations, which shrinks r_k toward 0 at long lags
$1.96/\sqrt{T}$: sd of r_k under white noise is about 1/sqrt(T); the band is 0.25 at T = 60, 0.16 at T = 144

r_k estimates the process property rho_k from one realization, so it carries sampling noise. The dashed band is that noise when the true correlation is zero: about 1 bar in 20 leaves it on pure noise, and 24 bars put 1.2 outside on average.

Worked by hand: six weekly defect counts (illustrative) Table

t	y_t	y_t - 6	lag-1 product	lag-2 product
1	2	-4	-	-
2	4	-2	(-2)(-4) = +8	-
3	5	-1	(-1)(-2) = +2	(-1)(-4) = +4
4	7	+1	(+1)(-1) = -1	(+1)(-2) = -2
5	8	+2	(+2)(+1) = +2	(+2)(-1) = -2
6	10	+4	(+4)(+2) = +8	(+4)(+1) = +4
sum	-	squares: 42	19	4

A factory's weekly defect counts, mean 6. r_1 = 19/42 = 0.452 and r_2 = 4/42 = 0.095; statsmodels acf returns exactly these two values. Redo it on paper: deviations, neighbour products, one shared denominator.

ACF of an AR(1) process with phi = 0.7 (exact values) Chart

Lag 1the only PACF bar of this process0.7
Lag 20.49
Lag 30.343
Lag 40.24
Lag 50.168
Lag 6inside the band for T = 144 (0.16)0.118

The ACF at lag k is phi to the power k, a geometric decay. The PACF of the same process is 0.7 at lag 1 and zero after it.

Why AR(1) decays geometrically: rho_k = phi^k, derived Formula

y_t = \phi\,y_{t-1} + \varepsilon_t \;\Rightarrow\; \gamma_k = \phi\,\gamma_{k-1} \;\Rightarrow\; \rho_k = \phi^{k}

$\gamma_k$: multiply both sides by y_(t-k), take expectations; the shock is uncorrelated with the past, so its term drops
$\rho_k$: divide the recursion by the variance gamma_0: rho_k = phi rho_(k-1), started at rho_0 = 1, solves to phi^k
$|\phi| < 1$: the stationarity condition; at phi = 1 the decay stops and the process is Day 21's random walk

Each extra lag multiplies the correlation by phi once more - the bars above, 0.7, 0.49, 0.343. The PACF regresses the in-between lags away instead, leaving the single direct bar at lag 1.

Bartlett's band widens after the first spikes Formula

\operatorname{Var}(r_k) \approx \frac{1}{T}\left(1 + 2\sum_{j=1}^{k-1} r_j^{2}\right)

$r_j,\ j<k$: earlier sample autocorrelations: each spike widens the band for every later lag
$1/T$: no earlier spikes gives back the flat white-noise band, 1.96/sqrt(T)
$MA(q)$: bars beyond lag q are judged against this wider band, not the flat one

statsmodels' acf returns these intervals when you pass alpha; the flat dashed band is the white-noise special case. Everything on this page - bands, Ljung-Box, signatures - separates estimation noise from structure.

Reading signatures, and turning them into candidates3 minWhy AR cuts off in the PACF and MA in the ACF, and a flow from bars to a first model.

The PACF is a regression coefficient Formula

y_t = \phi_{k1}\, y_{t-1} + \dots + \phi_{kk}\, y_{t-k} + e_t, \qquad \text{PACF}(k) = \phi_{kk}

$\phi_{kk}$: the coefficient on lag k once lags 1 to k-1 are in the regression
$AR(p)$: lag p is the last lag with a direct effect, so phi_kk is zero for k > p: the cut-off
$MA(q)$: has an infinite AR form, so phi_kk never cuts off: it decays

statsmodels' pacf defaults to method 'ywadjusted' (Yule-Walker); the course code uses Durbin-Levinson on the sample ACF. Both give the same picture on long series.

From the bars to a first candidate Diagram

Roughly stationary?after the ladder
- no: to Transform and difference first
yes
PACF cuts off at p?
- yes: to Try AR(p)
no
ACF cuts off at q?
- yes: to Try MA(q)
no
Both decay: ARMAlet an AICc search choose
- to Residuals, then backtest
Transform and difference firstthe ladder
Try AR(p)
- to Residuals, then backtest
Try MA(q)
Residuals, then backtestDays 23 and 25

Read lags 1-3 for the short memory and m, 2m for the season, separately: hourly electricity load carries lag-1 memory and a lag-24 spike at once. Mixed ARMA shows no clean cut-off in either plot, which is where an information-criterion search earns its keep (Day 23). The bars propose candidates; they never certify one.

Reading bars against the band3 minHow often single bars and the joint test fire on pure noise, and when the band misleads.

How often the reading rules fire (hands-on lab, 1,000 simulations each) Chart

Noise, T=60: a bar outside46 %
Noise, T=240: a bar outsidesingle bars cry wolf63 %
Noise, T=60: Ljung-Box rejects7 %
Noise, T=240: Ljung-Box rejectsthe joint test holds about 5%4 %
AR(1) 0.3, T=60: LB rejects32 %
AR(1) 0.3, T=240: LB rejectspower grows with length91 %

Share of simulated series flagged by "at least one of 24 bars outside the band" and by Ljung-Box with 10 lags at 5%. The band is the sampling noise of r_k under independence; on a trending series every bar is large, and for residuals of a fitted model or a short series it is only a rough guide.

The Ljung-Box test, and what passing means3 minThe statistic, its degrees of freedom, a hand computation, and a model that passes yet forecasts badly.

The Ljung-Box statistic Formula

Q = T(T+2)\sum_{k=1}^{h}\frac{r_k^{2}}{T-k}\;\sim\;\chi^{2}_{\,h-\text{model df}}

$h$: lags tested: about 10 for non-seasonal data, 2m for seasonal, at most T/5
$r_k$: residual autocorrelation at lag k
$\text{model df}$: estimated ARMA parameters, subtracted from the degrees of freedom
$\frac{T(T+2)}{T-k}$: small-sample weights that inflate later lags: what Ljung and Box added to Box-Pierce, which uses plain T

Applied to residuals of a fitted model. Today's airline model leaves 131 residuals (cap T/5 = 26, so 24 lags are allowed) and has two MA parameters, so 24 lags give 22 degrees of freedom.

Ljung-Box by hand on the six defect counts (h = 2) Table

Step	Arithmetic	Value
Inputs	r_1 = 0.452, r_2 = 0.095, T = 6	-
Lag-1 term	0.452^2 / (6 - 1)	0.0409
Lag-2 term	0.095^2 / (6 - 2)	0.0023
Q	T(T+2) x sum = 48 x 0.0432	2.07
p-value	chi-square with 2 df (raw series, no fitted model)	0.35
Box-Pierce, for contrast	T x (r_1^2 + r_2^2) = 6 x 0.214	1.28 (p = 0.53)

statsmodels' acorr_ljungbox returns the same Q and p. Six points cannot reject anything; the arithmetic is the point. The (T+2)/(T-k) weights make Q larger than Box-Pierce on short series.

Clean residuals, broken forecast (hands-on lab) Chart

In-sample one-step errorresidual Ljung-Box p = 0.620.82 MAE
Next 120 points, break at point 200a level shift of +8 the model never saw3.76 MAE

An AR(1) fitted on 180 points passes the residual test; a break at point 200 multiplies the error by more than four.

Differencing without overdoing it3 minThe spread criterion and the signature of one difference too many.

One step at a time on 12 years of monthly passengers (today's run) Table

Step	Lag-1 ACF	Lag-12 ACF	Std dev
log	+0.94	+0.79	0.352
log, d = 1	+0.55	+0.82	0.104
log, D = 1	+0.66	-0.39	0.059
log, D = 1, d = 1	-0.50	-0.54	0.049 (smallest)
log, D = 1, d = 2	-0.76	-0.55	0.085 (larger again)

Synthetic data, seed 22; the white-noise band is plus or minus 0.16. The raw series has almost the same ACF as the log (lag 1 +0.94, lag 12 +0.80) but a standard deviation in passengers (61.5).

Judge the last difference by two symptoms, never by the lag-1 bar alone Hierarchy

After a differenceread lag-1 ACF and the spread
- lag 1 near +1Under-differencedslow decay: one more difference
- lag 1 ≤ -0.5Spread fell?sd against the previous step
  - yesKeep: MA patternD=1, d=1: lag 1 -0.50, sd 0.049
  - no, it grewOver-differencedD=1, d=2: lag 1 -0.76, sd 0.085

Numbers from today's run (table above). At D = 1, d = 1 the lag-1 value is -0.50 yet the spread is the smallest: that is the moving-average pattern ARIMA(0,1,1)(0,1,1) models, not one difference too many.

Logs, Box-Cox and the way back4 minThe Box-Cox algebra, its inverse, and a forecast walked back to the money scale by hand.

The Box-Cox family Formula

w_t = \begin{cases} \log y_t & \lambda = 0 \\ (y_t^{\lambda} - 1)/\lambda & \lambda \neq 0 \end{cases}

$\lambda$: the transformation strength: 1 shifts only, 0.5 is a scaled and shifted square root, 0 is the log
$y_t > 0$: needed for the log and for most values of lambda

Choose the lambda that makes the seasonal swing about the same size at every level, then keep it simple; a log is easy to explain. Back-transformed log forecasts stay positive, which suits bookings, revenue and demand. Zeros break the log: log(y + c) distorts small counts, so use a count model (Day 26).

Back-transformation bias for a log forecast Formula

\operatorname{median}(y) = e^{m}, \qquad \mathbb{E}[y] = e^{\,m + \sigma^{2}/2} \approx e^{m}\left(1 + \tfrac{\sigma^{2}}{2}\right)

$m$: mean of the forecast on the log scale
$\sigma^{2}$: forecast variance on the log scale; it grows with the horizon

Exact when log-scale errors are normal. Use the mean for totals and budgets, the median for a typical single day.

Worked back-transform: a web shop's log-scale revenue forecast (illustrative) Table

Quantity	On the log scale	Back on the money scale
Point forecast m	ln(80) = 4.382	exp(4.382) = 80.0 - the median
Forecast sd	0.4 (variance 0.16)	-
Mean, bias-adjusted	m + 0.16 / 2 = 4.462	exp(4.462) = 86.7 - what totals need
80% interval	4.382 +- 1.282 x 0.4	[47.9, 133.6] - back-transform each end
Box-Cox, lambda = 0.5	w = (sqrt(y) - 1) / 0.5	y = (0.5 w + 1)^2 - invert the algebra; w = 14 gives back y = 64

Verified in Python: 1,000,000 log-normal draws with these parameters (numpy default_rng(0)) give median 80.0 and mean 86.7; the exact mean is 86.66. Intervals and medians back-transform directly; only the mean needs the adjustment, which matters once store forecasts are summed.

Median and mean part ways after a log Illustration

Skewed to the right: exp(mean of logs) = 100 is the median, the mean is 113.3 and the most likely value only 77.9. Totals need the mean.

With a one-step log sd of 0.07 the gap is 3.0% at 12 months and 6.1% at 24; noisier series reach double digits. Summing medians over routes understates the total by about that much.

Computed from the log-normal formulas: left, log-scale mean ln 100 and sd 0.5; right, log random walks with one-step sd 0.04, 0.07 and 0.12, so the h-step sd is sd√h and the gap is exp(h sd²/2) - 1. No data are simulated.

The back-transformed point forecast is the median; the mean sits above it, and the gap grows with the horizon because the log-scale variance does.

Decomposition with STL2 minTrend, seasonal and remainder layers, robust fitting and seasonal adjustment.

STL: a series as three layers Stack

ObservedLog it first if the season is multiplicative.
TrendA slow loess curve; the trend window sets how slowly it may bend.
SeasonalRepeats each period and may evolve slowly; the seasonal window sets how fast.
RemainderShould look like noise; robust fitting stops outliers from leaking into the other layers.

Subtracting the seasonal layer is seasonal adjustment. In the lab, on a logged multiplicative series, the peak-to-trough swing falls from 124.1 to 18.7.

Why autocorrelation breaks iid toolsadvanced2 minStandard errors, shuffled cross-validation and in-sample fit under serial dependence.

What serial dependence does to familiar tools Comparison

Criterion	Independent rows	Autocorrelated series
Standard errors	Valid as computed	Too small: neighbours repeat information
Shuffled cross-validation	An honest estimate	Optimistic: test days sit between training days (Day 25)
A clean fit	Some evidence of skill	Little evidence: fit the past, not the horizon

This is why Day 25 replaces shuffled k-fold with rolling origins for forecasting questions.

How many independent observations an AR(1) series is worth Formula

T_{\text{eff}} \approx T\,\frac{1-\phi}{1+\phi}, \qquad \phi = 0.7,\ T = 240 \;\Rightarrow\; T_{\text{eff}} \approx 42

$\phi$: lag-1 autocorrelation of an AR(1) process
$T_{\text{eff}}$: the sample size that gives the same variance of the mean with independent data

Naive standard errors of a mean treat 240 such points as 240; they carry about the information of 42. Standard errors come out roughly 2.4 times too small.

Memory anchor

Transform for the swings, difference as little as needed, read bars jointly - and clean residuals only mean nothing is left in the past.

Useful for

Preparing a series for ARIMA-type models (Day 23) and choosing seasonal and ordinary differencing with evidence.
Checking the residuals of any forecaster for left-over structure, from a seasonal naive baseline to a boosted model.
Producing a seasonally adjusted series for reporting or as an input feature.

Not suitable for

Proving that a model forecasts well - residual diagnostics describe the fit to the past and do not replace a backtest (Day 25).
Short or very noisy series, where ACF bars are too variable to identify orders; use information criteria and a backtest instead.

Traps

Differencing until the plot is clean

Each extra difference changes the ACF, and the plot can always be made to look tidier. Over-differenced series carry a negative lag-1 correlation and larger variance, and their forecasts have wider, less accurate intervals.

How to avoid it

Difference as little as the evidence supports (variance, KPSS sequence, lag-1 ACF) and confirm the choice with a backtest.

Reading a clean ACF as validation

White residuals show that the fitted model used up the linear dependence in the past sample. A level break, a new regime or an omitted driver can still sit in the future, and the diagnostic cannot see it.

How to avoid it

Treat residual checks as a filter and decide by out-of-sample error against a seasonal naive baseline.

Senior reasoning

A senior uses ACF and PACF to narrow candidates, not to name a model. They pick the transformation from how the swings scale, difference as little as the evidence supports, and test residuals jointly with the right degrees of freedom. They watch the scale when forecasts are summed, keep the median and the mean apart, and refuse to call a model validated until it has beaten the baseline on rolling origins.

Interview question

"Your model's residual ACF looks clean and Ljung-Box p is 0.4. Is the model good?" Structure the answer.

Outline your answer first.

Recall cards scheduled after completion

What do the ACF and PACF of AR(1), MA(1), a season and a random walk look like, and how much of the band is chance?
How do you detect over-differencing, and what does exp() of a log forecast return?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Forecasting: Principles and Practice (3rd edition) — Rob J Hyndman, George Athanasopoulos, 2021core
The white-noise definition and the plus-or-minus 1.96 over root T band; residual diagnostics and Ljung-Box with lag 10 or 2m and at most T/5; why too many differences induce false dynamics and how a KPSS sequence chooses d; back-transformation gives the median and needs a bias adjustment for the mean; STL decomposition. (Sections 2.9, 3.6, 5.4, 5.6 and 9.1)
On a measure of lack of fit in time series models — G. M. Ljung, G. E. P. Box, 1978optional
The portmanteau statistic of residual autocorrelations used as a joint lack-of-fit test.
statsmodels.stats.diagnostic.acorr_ljungbox (statsmodels documentation) — statsmodels developersoptional
The acorr_ljungbox arguments used in the code - lags, model_df for fitted-model residuals, and period.
statsmodels.tsa.stattools.acf (statsmodels documentation) — statsmodels developersoptional
The acf function used in the code - lag 0 is included in the output and the optional intervals use Bartlett's formula rather than the flat white-noise band.
statsmodels.tsa.seasonal.STL (statsmodels documentation) — statsmodels developersoptional
STL arguments used in the lab - period, seasonal and the robust flag that limits the influence of outliers.
statsmodels.tsa.statespace.sarimax.SARIMAX, fit and SARIMAXResults (statsmodels documentation) — statsmodels developersoptional
SARIMAX order and seasonal_order arguments and the resid attribute used for the residual check.
An Analysis of Transformations — G. E. P. Box, D. R. Cox, 1964optional
The power-transformation family with a likelihood-based choice of lambda; log at lambda 0.
STL: A Seasonal-Trend Decomposition Procedure Based on Loess — Robert B. Cleveland, William S. Cleveland, Jean E. McRae, Irma Terpenning, 1990optional
The STL procedure - iterated loess smoothing into trend, seasonal and remainder with robustness weights.

Optional sections

Deep dive · 10 min

Sample ACF. With mean $\bar y$ , the lag-k sample autocorrelation is $r_k = \sum_{t=k+1}^{T}(y_t-\bar y)(y_{t-k}-\bar y) / \sum_{t=1}^{T}(y_t-\bar y)^2$ . The statsmodels acf returns lag 0 as well, and its optional confidence intervals use Bartlett's formula, which widens with lag and differs from the flat white-noise band.

PACF. The partial autocorrelation at lag k is the coefficient on $y_{t-k}$ in a regression of $y_t$ on lags 1..k. For an AR(p) it is zero beyond p, which gives the cut-off; an MA process has an infinite AR representation, so its PACF decays.

Ljung-Box degrees of freedom. For residuals of an ARMA(p, q) fit the statistic is compared with $\chi^2$ on $h - p - q$ degrees of freedom; statsmodels exposes this as model_df. If h minus model_df is not positive the function returns NaN.

Box-Cox and the mean. Box-Cox with $\lambda = 0.5$ sits between no transformation and log. Bias adjustment for a log transform follows from the log-normal mean $e^{m+\sigma^2/2}$ ; the Box-Cox version depends on $\lambda$ as well. Zeros are a problem for the log: $\log(y+1)$ distorts small counts, so low-count series need a count model (Day 26), and bounded series use a scaled logit.

Why shuffling fails. With AR(1) dependence $\phi$ , the variance of a sample mean is about $(\sigma^2/T)\,(1+\phi)/(1-\phi)$ , so the effective number of independent observations is smaller than T and naive standard errors are too small. Shuffled k-fold CV additionally puts neighbours of each test point in training (Day 25).

STL choices. The seasonal window controls how fast the seasonal shape may change; robust=True down-weights outliers such as a holiday spike so they land in the remainder. For several seasonal periods use MSTL (Day 21 deep dive). Seasonal adjustment is for reporting and for models that take an adjusted series; forecast the adjusted series and add the seasonal part back.

Hands-on lab: What the reading rules do on white noise, and what clean residuals miss · 20 min

Run the code and answer: (1) For white noise of length 60, how often does at least one of 24 bars leave the band, and how often does Ljung-Box reject? (2) How much does the length change the chance that Ljung-Box catches an AR(1) with phi 0.3? (3) The AR(1) residuals are clean (p = 0.62) yet the one-step error after the break is far larger than in-sample: which part of the workflow would have caught this? (4) Set the STL robust flag to False and add one outlier of +40 to y: what happens to the seasonal strength and the adjusted series?

import numpy as np
from scipy.signal import lfilter
from statsmodels.stats.diagnostic import acorr_ljungbox
from statsmodels.tsa.seasonal import STL
from statsmodels.tsa.stattools import acf

rng = np.random.default_rng(220)

# 1) How often do the usual reading rules fire on pure white noise, and how often do they catch AR(1) 0.3?
def rates(n, phi, reps=1000):
    any_out = lb_hit = 0
    for _ in range(reps):
        x = lfilter([1], [1, -phi], rng.normal(size=n))
        r = acf(x, nlags=24)[1:]
        any_out += bool((np.abs(r) > 1.96 / np.sqrt(n)).any())
        lb_hit += acorr_ljungbox(x, lags=[10]).iloc[0, 1] < 0.05
    return any_out / reps, lb_hit / reps

for n, phi in [(60, 0.0), (240, 0.0), (60, 0.3), (240, 0.3)]:
    a, b = rates(n, phi)
    print(f"n = {n:3d}, phi = {phi}: >=1 of 24 lags outside band {a:4.0%}; Ljung-Box(10) rejects {b:4.0%}")

# 2) Clean residuals do not prove the forecast is suitable: a level break after the fitting window
x = lfilter([1], [1, -0.5], rng.normal(size=300)) + 50
x[200:] += 8                                         # break the model has never seen
train, test = x[:180], x[180:]
d = train - train.mean()
phi = (d[:-1] @ d[1:]) / (d[:-1] @ d[:-1])           # least-squares AR(1) coefficient
res = d[1:] - phi * d[:-1]
print(f"AR(1) on the first 180 points: phi = {phi:.2f}, residual Ljung-Box(10, model_df=1) p = "
      f"{acorr_ljungbox(res, lags=[10], model_df=1).iloc[0, 1]:.2f}")
pred = train.mean() + phi * (x[179:-1] - train.mean())   # one-step forecasts for t = 180..299, parameters frozen
print(f"one-step MAE: in-sample {np.abs(res).mean():.2f}; next 120 points {np.abs(test - pred).mean():.2f}")

# 3) STL on the log of a multiplicative seasonal series, then seasonal adjustment
t = np.arange(144)
y = np.exp(np.log(100) + 0.008 * t + 0.25 * np.sin(2 * np.pi * t / 12) + rng.normal(0, 0.03, 144))
stl = STL(np.log(y), period=12, robust=True).fit()
strength = max(0.0, 1 - stl.resid.var() / (stl.resid + stl.seasonal).var())
adjusted = np.exp(np.log(y) - stl.seasonal)
print(f"seasonal strength (1 - var(R) / var(S + R)): {strength:.2f}")
print(f"peak-to-trough swing: raw {np.ptp(y[-12:]):.1f}, seasonally adjusted {np.ptp(adjusted[-12:]):.1f}")

Interview practice · 10 min

A retailer's demand team fits an ARIMA model to store-level sales, shows a clean residual ACF and Ljung-Box p of 0.3, and says the model is ready for the replenishment plan. A colleague on the same team had earlier logged the series, and the plan sums the store forecasts into one warehouse order. What do you say?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-18 · depends on statsmodels, numpy, scipy

1 Recall · 5 min

2 Concept · 10 min