Neural networks, deep learning, NLP · insight-0445
Do not expect a plain RNN to carry information across hundreds of steps
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- energy, telecom
Summary: situation, action, reason, boundary
Situation
You model long sequences, such as months of daily meter readings or long support transcripts, with a vanilla RNN.
Action
Start with engineered lag and window features plus a GBM or linear model. If a sequence model is needed, use gated units (LSTM or GRU), truncate or summarize history, or use attention or convolutions over the window.
Reason
Long-term dependencies are hard for gradient descent through recurrence; gating mitigates this, and attention replaces sequential steps with direct connections between positions.
Boundary
Sequence tasks with long contexts and modest data.
Symptom or risk
Gradients vanish or explode over many time steps, so the model relies on recent inputs only; training is also slow because steps are processed one after another.
Exceptions
Short sequences, streaming settings with tight memory, or strong recent-history signals where small RNNs work well.
Trade-offs
Attention costs grow quadratically with window length; recurrent models stay linear in it.
Pitfalls
Feeding years of raw history and assuming the network will find the useful lags.
Lessons and concepts
Claims and their sources
Bengio, Simard and Frasconi showed learning long-term dependencies with gradient descent is difficult.
Hochreiter and Schmidhuber introduced the LSTM gated memory cell for long-range dependencies.
Vaswani et al. compare layer types, showing recurrent layers need O(n) sequential operations while self-attention needs O(1).