Neural networks, deep learning, NLP · insight-0449
Budget attention cost by sequence length before raising the context limit
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- manufacturing, aviation
Summary: situation, action, reason, boundary
Situation
Classifying long maintenance logs or contracts, someone proposes raising the maximum length from 512 to 4,096 tokens.
Action
Measure where the label signal sits first. Prefer truncation, chunking with aggregation, or retrieval of relevant sections; if long context is needed, use memory-efficient attention kernels and re-measure memory and latency.
Reason
Exact memory-efficient kernels such as FlashAttention reduce memory traffic, but they do not remove the quadratic number of pairwise scores.
Boundary
Transformers on long documents, logs or sequences.
Symptom or risk
Self-attention time and memory grow quadratically with length, so an 8x longer input costs about 64x more in the attention layers; jobs hit out-of-memory errors or latency limits.
Exceptions
Short inputs, where feed-forward layers, not attention, dominate cost.
Trade-offs
Chunking can split evidence across chunks; long context costs money and latency.
Pitfalls
Extrapolating from a short-input benchmark to long-input production latency.
Lessons and concepts
Claims and their sources
Vaswani et al. give self-attention per-layer complexity O(n^2 d) in sequence length n and representation dimension d.
Dao et al. note self-attention time and memory are quadratic in sequence length and present FlashAttention, an exact IO-aware attention algorithm that reduces GPU memory reads and writes.