Sequence Models
By the end: You trace a hidden state over four steps by hand, show how the gradient path through time shrinks, explain what teacher forcing hides, and say when a recurrent model is worth testing against classical forecasting baselines.
At a glance
A recurrent network overwrites one small state with the same rule at every step, so memory and gradients fade with distance; gates slow the fade.
Hidden state
A fixed-size summary, overwritten each step by the same learned rule.
Shared weights
Unrolled over T steps, the cell is a T-layer network with tied weights.
Fading gradient
A product of per-step factors below 1: 1.5e-9 after 30 steps here.
Gates
LSTM and GRU keep a mostly additive path; a 0.97 gate keeps 41% over 30 steps.
Baseline first
Seasonal naive and lag models, free-running, rolling origin, before an LSTM.
Recall
No notes: (1) the chain rule for and why a long chain of factors below 1 goes to zero (Day 33); (2) the seasonal naive forecast for next Tuesday (Day 23); (3) why a recursive 14-day forecast can drift further from the truth than a direct one (Day 24). Today connects all three.
Concept
A shopper's click stream, a sentence and a pump's vibration trace are ordered. A Recurrent network (RNN) reads one step at a time and carries a small vector forward, the Hidden state, like notes on a notepad of fixed size. The analogy breaks at the notepad: the same learned rule overwrites it at every step, so whatever the rule does not protect fades.
Predict first
With a recurrent weight of 0.5, how does the gradient reaching step 1 at 20 steps compare with 5 steps?
Text description of the figure
A two-unit recurrent cell unrolled over up to 20 steps, showing the hidden state at each step and the size of the gradient path from the last step back to each earlier step. Controls change the number of steps, the recurrent weight, the activation (tanh or linear) and the input pattern.
- x1 = 1a one-off spikew_x
- h1 = 0.762tanh(1 + 0.5 x 0)
- to h2
- x2 = 0w_x
- h2tanh(0 + 0.5 x h1)
- to h3
- x3 = 0w_x
- h3tanh(0 + 0.5 x h2)
- to h4 = 0.090
- x4 = 0w_x
- h4 = 0.09012% of h1 is left
Every horizontal arrow is the same weight w_h = 0.5; fill in h2 and h3 yourself in the application task. The spike is still visible at step 4, but fading.
Each step back multiplies by at most 0.5: step 0 receives 6.5e-10 of the signal.
Closer to 1 fades more slowly, but still fades: 0.018 after 30 steps.
Above 1 with nothing to squash it, the path explodes: 2.6e3. This is what clipping guards against.
A large weight does not rescue tanh: constant input saturates it, slopes near 0, and the path dies (1.3e-15).
- linear, one pulse
- tanh, one pulse
- tanh, input every step
- dashed: product of per-step norms (upper bound)
Linear cell: the 30-step path stays within a factor of 100 of 1 only for w between 0.90 and 1.15. tanh cannot explode, but it vanishes whenever it saturates.
Two hidden units, W_hh a scaled rotation, no random numbers: every curve is the exact Jacobian product computed by the RNN-unroll widget code. Values are clipped to 1e-20 and 1e8 for display.
Each panel changes one knob of the same two-unit cell. Below 1 the product dies, above 1 a linear cell explodes, and tanh trades explosion for saturation.
| Criterion | Vanilla RNN | LSTM | GRU |
|---|---|---|---|
| State carried | h only | h and a cell state c | h only (merged) |
| Gates | None | Forget, input, output, plus a candidate update | Update and reset, plus a candidate |
| Parameters (input d, hidden h) | h(d + h + 1) | 4h(d + h + 1) | 3h(d + h + 1), about three quarters of an LSTM |
| Long dependencies | Fades within tens of steps | Longer while the forget gate stays near 1 | Similar to LSTM on many tasks |
Counts assume one bias vector per gate; PyTorch keeps two, so its counts use d + h + 2. Gating helps; it does not make memory unlimited.
- vanilla, w = 0.5 (bound)
- forget gate 0.9
- forget gate 0.97
- forget gate 0.99
After 100 steps a 0.99 gate still keeps 37%; a 0.9 gate keeps 2.7e-5; the vanilla bound is below 1e-30 (off the chart). Dashed line: half left; each dot is that gate's half-life.
Memory half-life = ln 2 / (-ln f). Moving a gate from 0.9 to 0.99 multiplies the half-life by about ten; the network has to learn to hold f that close to 1.
Exact arithmetic, no data: f to the power k along the cell path. Real gates vary from step to step and are learned; the curves show what a gate value implies, not what a trained LSTM does.
Along the additive cell path a signal keeps f to the power k. The gate value converts directly into how long the cell can remember.
- teacher-forced (true history)
- free-running (own forecasts)
- seasonal naive
Teacher-forced MAE stays near 2.8 at every step because the true previous day is fed in. Free-running, the same model reaches 4.5 by day 14; seasonal naive sits at 4.4 on average.
One origin up close: teacher-forced forecasts are re-anchored to the truth every day; free-running ones carry yesterday's error forward and drift toward the model's long-run average.
Seeded synthetic daily series (weekly shape, trend, autocorrelated noise); one least-squares model on 7 lags, fitted on the first 440 days and scored on the rest. Same model in both modes: only the evaluation differs.
Feeding true history every step hides how the model's own errors compound over a 14-day plan.
- Vanilla cell: , same weights at every step; unrolled over steps it is a -layer network with tied weights.
- The gradient from step back to step 1 is a product of per-step factors: below 1 it vanishes (Vanishing gradient through time), above 1 it explodes (Bengio et al., 1994); clipping is the usual patch (Pascanu et al., 2013).
- LSTM and GRU cells add gates and a mostly additive path, so the signal survives longer; not forever.
- Decoders trained with Teacher forcing see true previous values; at inference they see their own, and errors compound (Exposure bias).
- For forecasting the bar is classical: seasonal naive, ETS and lag-feature models (Days 23-24, 27).
Assumptions
Order matters and history beyond a fixed window carries signal. If 14 lag features capture it, recurrence adds cost, not information.
The fixed-size state can compress what is needed. A long context with sparse key events may need attention or retrieval.
Training and deployment regimes match. Teacher forcing, padding and truncation are part of the model you evaluate.
Application and workplace decision
By hand first, with , , and :
- write to and the product that gives ;
- predict whether the factor over 30 steps is nearer or .
The code then:
- unrolls the cell and checks the analytic gradient against a finite difference;
- prints the factor for 4, 10 and 30 steps, and an additive cell path with forget gate 0.97.
The lab tests the forecasting side: seasonal naive, a linear lag model and a small neural net on lags.
Python
import numpy as np
w_x, w_h, b = 1.0, 0.5, 0.0 # one hidden unit; the same weights at every step
def unroll(x, h0=0.0):
h, out = h0, []
for x_t in x:
h = np.tanh(w_x * x_t + w_h * h + b)
out.append(h)
return np.array(out)
x = np.array([1.0, 0.0, 0.0, 0.0]) # one booking spike, then silence
h = unroll(x)
print("hidden state h1..h4:", np.round(h, 3))
def grad_h1(T):
"""d h_T / d h_1 = product over steps 2..T of w_h * (1 - h_t^2)."""
hs = unroll(np.r_[1.0, np.zeros(T - 1)])
return float(np.prod(w_h * (1 - hs[1:] ** 2)))
def tail(h1, steps=3): # steps 2..4 with zero input
for _ in range(steps):
h1 = np.tanh(w_h * h1)
return h1
eps = 1e-6
numeric = (tail(h[0] + eps) - tail(h[0] - eps)) / (2 * eps)
print(f"d h4/d h1: analytic {grad_h1(4):.4f}, numeric {numeric:.4f}")
for T in (4, 10, 30):
print(f"T={T:2d}: gradient factor from step 1 = {grad_h1(T):.2e}")
print(f"additive cell path, forget gate 0.97, 30 steps: {0.97 ** 29:.2f}")
Mini-assessment
Technical depth
The model, definedThe recursion every recurrent network computes, what is learned, and the assumptions written as properties of the equation.
- input at step t, a d-vector (today's sales and calendar flags, one second of sensor readings, a token embedding)
- the hidden state, an h-vector: the ONLY channel from steps 1..t to the output at t
- h x d, h x h and h numbers, the SAME at every step (tied weights)
- read-out from the state to the prediction, also shared across steps
- sequence length; it changes the computation, never the parameter count
Unroll the recursion and h_T is one deep composition of the same function T times. Everything the model knows about the past at step t must fit in the h numbers of h_t.
Learned
W_x, W_h, b and the read-out W_y, c. The state h_t is not learned: it is recomputed per sequence.
A sufficient state
The prediction at t sees x_1..x_t only through h_t; what the rule overwrites, no read-out recovers.
Time-invariant dynamics
The same W at every step. A store remodel or a sensor recalibration breaks it.
Objective
Sum of per-step losses, e.g. sum of (y_t - y_hat_t)^2, minimised through the unrolled graph (BPTT).
Size and cost
h(d + h + 1) numbers before the read-out, whatever T; about T x h(d + h) multiply-adds, strictly in order.
The recursion unrolled, by handFour steps of a one-unit cell on a telecom alarm sequence, the state and the gradient path written out cell by cell, and the saturation the numbers reveal.
| Step t | Input x_t | a_t = x_t + 0.8 h_(t-1) | h_t = tanh(a_t) | Slope 0.8 (1 - h_t^2) | d h_t / d h_1 |
|---|---|---|---|---|---|
| 1 | 1 | 1 + 0.8 x 0 = 1.000 | 0.762 | (not used) | 1 |
| 2 | 1 | 1 + 0.8 x 0.762 = 1.609 | 0.923 | 0.118 | 0.118 |
| 3 | 0 | 0 + 0.8 x 0.923 = 0.738 | 0.628 | 0.484 | 0.118 x 0.484 = 0.057 |
| 4 | 0 | 0 + 0.8 x 0.628 = 0.503 | 0.464 | 0.628 | 0.057 x 0.628 = 0.036 |
Synthetic telecom example: two consecutive alarm ticks on a cell tower, then silence. Verified in numpy, with a finite difference giving the same 0.036. The second tick saturates the unit (h = 0.923), so its slope collapses to 0.118: saturation cuts the path even though w_h = 0.8 is not small.
- one factor per step (steps 4, 3, 2 from left), each w_h x tanh'(a_t) = w_h (1 - h_t^2)
- the same 0.8 at every step
- depends on how hard the unit is driven: 0.148 at the saturated step 2
Redo it on paper from h_0 = 0 with the row rule. The application code repeats the computation with w_h = 0.5 and adds a finite-difference check.
Vanishing and exploding, derivedThe w-to-the-k computation on a linear cell, the general product of Jacobians and its bound, and the orders of magnitude a vanilla cell loses.
| w_h | k = 5 | k = 10 | k = 30 | Reading |
|---|---|---|---|---|
| 0.5 | 0.031 | 9.8e-4 | 9.3e-10 | Gone within ten steps |
| 0.9 | 0.59 | 0.35 | 0.042 | Slow fade: about 4% left at 30 steps |
| 1.0 | 1 | 1 | 1 | The knife edge: neither fades nor grows |
| 1.1 | 1.61 | 2.59 | 17.4 | Explodes slowly; long sequences hit spikes |
| 1.5 | 7.6 | 57.7 | 1.9e5 | Explodes fast: NaN without clipping |
With h_t = w h_(t-1) + x_t, each step multiplies d h_t / d h_(t-k) by w, so the factor is w^k. Check one cell on paper: 0.9^10 = 0.349. A matrix W_h behaves the same way along each eigen-direction, with |eigenvalue| playing w.
- the shared recurrent weight (0.5 here)
- tanh slope at step t: at most 1
- one factor per step: below 1 shrinks geometrically, above 1 explodes
- activation slope at step t: tanh slopes are at most 1 and near 0 when saturated
- largest singular value of the recurrent matrix (w in the demo)
- the bound is the dashed line in the demo; the true product can be smaller
Below 1 per step the product vanishes geometrically, above 1 it can explode (Bengio et al., 1994; Pascanu et al., 2013).
Values printed by the application code. Each extra vanilla step multiplies by at most w_h = 0.5.
Vanishes geometrically
Like w^k. A dependency 30 steps back gets almost no signal even if it matters.
Explodes
Rare huge norms on long sequences; clip the global norm (all gradients as one vector).
Vanishes fastest
Both factors small: the path is cut within a few steps.
Saturation caps it
tanh keeps the state bounded; a saturated unit passes almost nothing back.
Vanishing is a property of the path, not of the loss. Per step the factor is diag(f'(a_t)) W_h, and the chain rule multiplies T - k of them; PyTorch clip_grad_norm_ bounds the product's effect on the step.
The LSTM cell, gate by gateadvancedAll four blocks written out, one step computed by hand, why the additive path keeps gradients alive, and the parameter count derived.
- one affine map of [x_t, h_(t-1)] per block: W is h x d, U is h x h, b is h
- sigmoid, output in (0, 1): a soft switch per state unit
- forget gate: the fraction of each unit of c_(t-1) that is kept
- input gate times candidate: what is written (g in (-1, 1) via tanh)
- output gate: how much of tanh(c_t) is exposed as h_t
Parameter count follows directly: four blocks, each h(d + h) weights plus h biases, so 4h(d + h + 1). With d = 4 inputs and h = 32 units: 4 x 32 x 37 = 4,736. The window length never appears.
- forget gate in (0, 1): how much of the old cell state to keep
- input gate and candidate: what to write
- output gate: what to expose as h_t
- the additive path: gradients flow along c without a squashing step
Each gate is a sigmoid of W [x_t, h_{t-1}] + b; the candidate uses tanh. The gated cell is from Hochreiter and Schmidhuber (1997); the forget gate was added to it later.
Forget, z_f
w = 0.5, u = 0.5, b = 1: z_f = 0.5 + 0.25 + 1 = 1.75, so f = sigmoid(1.75) = 0.852
Input, z_i
w = 1, u = 0.5, b = 0: z_i = 1.25, so i = sigmoid(1.25) = 0.777
Candidate, z_g
w = 1, u = -0.5, b = 0: z_g = 0.75, so g = tanh(0.75) = 0.635
Output, z_o
w = 0.5, u = 0.5, b = 0: z_o = 0.75, so o = sigmoid(0.75) = 0.679
Cell state
c_t = f c_(t-1) + i g = 0.852 x 1.0 + 0.777 x 0.635 = 0.852 + 0.494 = 1.346
Hidden state
h_t = o tanh(c_t) = 0.679 x 0.873 = 0.593
Gradient along c
d c_t / d c_(t-1) = f = 0.852 directly: no tanh slope and no W_h in this factor
Same cell, b_f = 0
f = sigmoid(0.75) = 0.679; after 30 steps 0.679^30 is about 1e-5 against 0.852^30, about 8e-3
Hand-set weights, verified in numpy. The forget-gate bias decides how far the additive path reaches; a positive initial forget bias is a common practitioner choice for that reason. PyTorch initialises all biases uniformly near 0, so f starts near 0.5.
- c_(t-1)old cell state
- to x f_t
- x_t and h_(t-1)feed all four blocks
- to Gates f, i, o and g
- x f_tkeep a fraction
- to + i_t g_t
- Gates f, i, o and gsigmoids; tanh for g
- f_t: back to x f_t
i_t, g_t - + i_t g_twrite new content
- c_tto the next steptanh, x o_t
- h_t = o_t tanh(c_t)exposed state, to next step
Top row: d c_t / d c_(t-1) = f directly, no tanh slope and no W_h, against w(1 - h^2) for a vanilla cell. f = 0.97 keeps 41% over 30 steps, 0.85 under 1%. Indirect terms through h_(t-1) feeding the gates exist but are usually smaller: a highway, not a guarantee.
Properties, failure modes and their diagnosticsWhat a recurrent model guarantees and assumes, how each property breaks, the cheap check for each, and a decision flow from symptom to fix.
| Property | What it means | How it breaks | Diagnostic |
|---|---|---|---|
| Tied weights | One rule for every step; any length T | Dynamics change over time (regime shift) | Error by period on a rolling-origin backtest |
| Fixed-size state | History compressed into h numbers | Sparse key events far back get overwritten | Planted early cue: can the model use it at all? |
| Gradient = product of T - 1 factors | Signal from step 1 scales like w^(T-1) | Vanishes (|w| < 1) or explodes (|w| > 1) | Log the global gradient norm per step |
| Saturation | tanh slope near 0 when |a| is large | Path dies even with w near 1 | Histogram of |h|; share above 0.95 |
| Truncated BPTT, length K | Gradients flow K steps back at most | Dependencies longer than K cannot be learned | Retrain with 2K and compare |
| Sequential compute | Step t waits for step t - 1 | Slow training on long sequences | Time per epoch against sequence length |
| Teacher-forced training | Trained on true previous values | Errors compound free-running (exposure bias) | Free-running error per horizon |
Training memory grows with T because every h_t is stored for the backward pass; compute per sequence is about T x h(d + h) multiply-adds per block. The diagnostics are cheap; run them before tuning hidden size.
- Loss spikes or NaN?log gradient norm
- no: to Early signal unused?
yes - Clip the global normthen lower the learning rate
- Early signal unused?planted early cue
- no: to Beats a lag model?
yes - Gated cell, shorter pathLSTM/GRU, attention, explicit lags
- Beats a lag model?same backtest
- yes: to Keep as a candidate
no - Keep the lag modelcheaper to run and explain
- Keep as a candidatefree-running, per horizon, seeds
Check in this order, cheapest first. A planted cue (a synthetic early spike that decides the target) tells you whether the model can carry information that far at all.
| Criterion | Memory (forward pass) | Learnability (backward pass) |
|---|---|---|
| The question | Does h_t still carry the early event? | Does the gradient reach that far back? |
| Inspect | The state h_t in the steps after the cue | Per-step gradient norms |
| Do gates help? | Yes, while f stays near 1; never unlimited | Yes, same condition; never unlimited |
A planted cue tests both: in a synthetic copy of a SaaS session log, make the target depend on an event 50 steps back and see whether any architecture learns it.
The knobs and their defaultsadvancedHidden size, depth, dropout, layout, direction, truncation and clipping, and what each trades.
| Knob | Default | Trade-off |
|---|---|---|
| hidden_size | set by you | Capacity of the state vs parameters (4h(d + h + 2) per LSTM layer) |
| num_layers | 1 | Deeper features vs a longer gradient path and more seed variance |
| dropout | 0 (between stacked layers only) | Regularises stacks; does nothing with one layer |
| batch_first | False: (seq, batch, feature) | A silent shape bug when your data is (batch, seq, feature) |
| bidirectional | False | Reads the future: fine for tagging, a leak for forecasting |
| Truncation length (BPTT) | your training loop | Memory and speed vs the longest learnable dependency |
| Gradient clipping | none unless you call it | Bounds exploding steps; hides a learning rate that is too high |
Defaults read from the PyTorch 2.14 nn.LSTM page (two bias vectors per layer, each 4 x hidden; weights uniform in plus or minus 1 / sqrt(hidden)).
Sequence models for forecastingThe baselines, the two evaluation modes and the training tricks that narrow the gap between them.
Seasonal naive
Same weekday last week: the floor every model must clear (Day 23)
ETS or a lag-feature model
Ridge or boosting on 7-28 lags plus calendar features (Day 24)
Rolling-origin backtest
Many origins, error per horizon, free-running (Day 25)
Neural candidate
A small MLP on lags first; an RNN only if long history carries signal
Seeds and combination
Report spread; a combination often beats any single model (Day 26)
| Criterion | Teacher forcing | Free-running | Scheduled sampling | Direct multi-horizon head |
|---|---|---|---|---|
| Input at each step | The true previous value | The model's own previous output | True values early in training, then more of its own | History only; all 14 days predicted at once |
| What it gives you | One-step-ahead skill | The multi-step forecast you will deploy | A training scheme (Bengio et al., 2015) | Each horizon scored directly (Day 24) |
| Hides or fixes | Hides how early mistakes compound over the horizon | Hides nothing: the number to report per horizon | Narrows the gap; still score free-running | No feedback loop, so no errors to compound |
A model that looks excellent one step ahead can still lose to a simple baseline over a multi-step horizon.
Why attention took over, and the neighboursThree structural limits of recurrence and how lag models, dilated convolutions and attention compare.
Sequential
Step t needs step t - 1, so training cannot run in parallel over time.
Bottleneck
The whole past must pass through one fixed-size state.
Path length
Two positions 100 steps apart are 100 multiplications apart.
Attention
Every position reads every other directly (Day 39), at n squared cost.
| Criterion | Lag features + GBM or ridge | RNN, LSTM, GRU | Dilated 1-D convolutions | Attention (Day 39) |
|---|---|---|---|---|
| How the past reaches now | Explicit window you choose | One state, updated step by step | Fixed receptive field (Day 36) | Every position reads every other |
| Parallel training | Yes | No, sequential over time | Yes | Yes, at n squared cost |
| Typical home | Demand forecasting baselines (Days 24, 27) | Streaming, small state, low latency | Long windows of sensor or audio data | Language; long context with sparse key events |
Padding, truncation and clippingImplementation details that silently change what the model learns.
Padding is input too
Pad values pass through the cell like real data; know which step your read-out uses.
Truncated backpropagation
Gradients flow only within each chunk; longer dependencies cannot be learned.
Exploding gradients
Clip the global norm; check the learning rate first.
Batch dependence
A series' forecast must not change with what it is batched with.
What a staff-level answer addsadvancedNuances and misreadings that separate an experienced answer from a textbook one.
| Criterion | Common reading | Staff-level reading |
|---|---|---|
| Can it remember? | 'The state remembers the sequence' | Separate memory from learnability; gates help both, neither is unlimited |
| Reporting a number | An error with no evaluation mode named | One-step teacher-forced and free-running multi-step answer different questions |
| LSTMs and vanishing | 'LSTMs solved vanishing gradients' | They slowed the decay along the cell path; the gate must stay near 1 |
| More history | 'More history always helps' | Without a mechanism that reaches it, history is cost; test with a planted cue |
| The close | 'It wins on the metric' | Beat seasonal naive and a lag model by more than the seed spread |
The cost-aware close matters because a recurrent model's training and monitoring are paid for every retrain.
Memory anchor
Same weights each step, gradients multiply each step: gate it, shorten it, or attend - and beat seasonal naive first.
Useful for
Reading a colleague's sequence-model proposal and asking the three questions that matter (state size, evaluation mode, baseline).
Understanding why gated cells, attention and decoders appear in the same family tree before Day 39.
Deciding when a recurrent or sequence model is worth testing on a forecasting or event-sequence problem (store demand, card-transaction streams, SaaS session logs, machine telemetry).
Not suitable for
Establishing that a recurrent model will beat a classical forecaster on your data - only a rolling-origin comparison can.
Dependencies hundreds of steps apart where a plain recurrent cell is the proposed solution.
Training a production recurrent model - this day gives the mechanism and the limits, not a training recipe.
Traps
Validating with teacher forcing and deploying free-running
One-step error with true history looks excellent while multi-step forecasts, which consume the model's own outputs, drift. The reported number answers a question the business is not asking.
How to avoid it
Evaluate in the mode and at the horizons used in production, per horizon, on rolling origins (Day 25).
A sequence model against a weak baseline
Beating a naive last-value forecast or one backtest origin proves little. Seasonal naive, ETS and lag-feature linear or boosted models are strong for many demand series and are cheap to maintain.
How to avoid it
Put seasonal naive, a lag model and the sequence model through the same backtest and metric, and report spread over seeds.
Senior reasoning
A senior asks what the state has to remember and for how long (a weekly cycle in store sales, a fraud pattern spread over a card's last 50 transactions, a slow drift in a turbine's telemetry), then asks for the evaluation that matches deployment: free-running, per horizon, rolling origin. They treat a recurrent model as a candidate that must beat a seasonal naive and a lag-feature model on cost-adjusted terms, and they know when to reach for attention, direct multi-horizon heads or a combination instead of a deeper recurrent stack.
Interview question
"Why did attention-based models replace LSTMs for many language tasks, and would you still use an LSTM?"
Recall cards scheduled after completion
- Why does the gradient from early steps vanish in a plain recurrent network, and what does an LSTM change?
- What is exposure bias, and how do you evaluate for it?
Prerequisites and related concepts
References
- Learning long-term dependencies with gradient descent is difficultcore
Formal account of why gradient descent has difficulty learning long-term dependencies through repeated multiplication in recurrent networks.
- Long Short-Term Memorycore
Introduces the gated memory cell (LSTM) designed for long-range dependencies.
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationoptional
Introduces the RNN encoder-decoder and the gated recurrent unit (GRU).
- Sequence to Sequence Learning with Neural Networksoptional
LSTM encoder-decoder for sequence transduction with a fixed-size vector between encoder and decoder.
- Neural Machine Translation by Jointly Learning to Align and Translateoptional
Attention over encoder states removes the fixed-length bottleneck of the plain encoder-decoder.
- On the difficulty of training recurrent neural networksoptional
Analysis of the exploding and vanishing gradient problems in recurrent networks and gradient-norm clipping as a remedy.
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networksoptional
Training that gradually feeds the model its own previous outputs to narrow the gap between teacher forcing and free-running use.
- torch.nn.utils.clip_grad_norm_ (PyTorch documentation)optional
Clipping the global gradient norm, computed as if all gradients were one vector.
- The M4 Competition: 100,000 time series and 61 forecasting methodsoptional
Results paper of a large forecasting competition comparing statistical, machine-learning, combination and hybrid methods; read its findings section before quoting rankings.
- Are Transformers Effective for Time Series Forecasting?optional
Simple linear models outperformed several Transformer forecasting models on standard long-horizon benchmarks, a reminder to run baselines.
Optional sections
Deep dive
Backpropagation through time. Unroll the cell, treat each copy as a layer with tied weights, and apply the chain rule. The gradient of the loss at step with respect to is a product of Jacobians . Its size is governed by the largest singular value of and the tanh slopes. Truncated BPTT cuts the chain after a fixed number of steps for memory and speed, which also caps how far a dependency can be learned.
Parameter counts. For input size and hidden size , a vanilla cell has parameters, an LSTM four times that (four gate blocks), a GRU three times. With , : 1,184, 4,736 and 3,552. Library implementations may keep two bias vectors per block, which adds a few more.
Why exposure bias is not only a neural problem. A recursive forecast from a one-step linear AR model compounds in the same way; direct multi-horizon models trade a larger number of models (or outputs) for no feedback loop (Day 24). Scheduled sampling and similar training schemes mix the model's own predictions into training to narrow the mismatch.
Where recurrent models still make sense. Streaming settings with a small state and strict latency, event sequences with irregular timestamps, and series with many related sequences and rich covariates can justify a global recurrent model. Each case still needs the baseline comparison from Days 23-27.
Hands-on lab: Seasonal naive vs linear lags vs a small neural net, direct and recursive (synthetic)
One synthetic daily series with a weekly pattern, trend and autocorrelated noise. Models: seasonal naive, ridge on 14 lags predicting all 14 horizons directly, a small MLP on the same lags (a stand-in for a neural model, not an LSTM), and a one-step ridge applied recursively on its own outputs. Rolling origins every 7 days. Read the error by horizon, then change the noise autocorrelation from 0.6 to 0.9 and say which model's ranking moves and why this single synthetic series cannot tell you how an LSTM would do on real store, load or traffic series.
import numpy as np
from sklearn.linear_model import Ridge
from sklearn.neural_network import MLPRegressor
rng = np.random.default_rng(37)
t = np.arange(760)
season = np.array([0, -4, -6, -2, 6, 14, 10])[t % 7] # weekly shape (synthetic passengers)
noise = np.zeros(len(t))
for i in range(1, len(t)):
noise[i] = 0.6 * noise[i - 1] + rng.normal(0, 3) # autocorrelated noise
y = 120 + 0.03 * t + season + noise
L, H = 14, 14 # 14 lags in, 14 days out
def windows(series, start, stop):
X = np.array([series[i - L:i] for i in range(start, stop)])
Y = np.array([series[i:i + H] for i in range(start, stop)])
return X, Y
split = 600
Xtr, Ytr = windows(y, L, split - H + 1)
direct = Ridge(alpha=1.0).fit(Xtr, Ytr) # one linear model, 14 outputs
one_step = Ridge(alpha=1.0).fit(Xtr, Ytr[:, 0]) # applied recursively below
mlp = MLPRegressor(hidden_layer_sizes=(32,), max_iter=800, random_state=0).fit(Xtr, Ytr)
def recursive(model, hist):
h, out = list(hist), []
for _ in range(H):
nxt = model.predict(np.array(h[-L:])[None])[0]
out.append(nxt)
h.append(nxt)
return np.array(out)
names = ("seasonal naive", "ridge direct", "mlp direct", "ridge recursive")
err = {k: [] for k in names}
for o in range(split, len(y) - H, 7): # rolling origin, every 7 days
hist, truth = y[o - L:o], y[o:o + H]
naive = np.array([y[o - 7 + (j % 7)] for j in range(H)])
preds = [naive, direct.predict(hist[None])[0], mlp.predict(hist[None])[0], recursive(one_step, hist)]
for k, p in zip(names, preds):
err[k].append(np.abs(p - truth))
for k, v in err.items():
v = np.array(v)
print(f"{k:16s} MAE h=1 {v[:, 0].mean():5.2f} h=7 {v[:, 6].mean():5.2f} h=14 {v[:, 13].mean():5.2f} all {v.mean():5.2f}")
Interview practice
At a regional energy utility, a teammate says their LSTM gives "MAE 2.1 MW" on next-day load and wants to ship it for the 14-day dispatch plan. What do you ask before agreeing?