Neural networks, deep learning, NLP · insight-1464
Test on sequences longer than any seen in training before assuming the position scheme extends
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A model trained on sessions of up to 128 events will see sessions of 400 events in production.
Action
Build a length-stratified test set that includes lengths 2 to 4 times the training maximum, plot the metric against length, and truncate, chunk or retrain with a scheme made for extrapolation if it falls.
Reason
Hugging Face's glossary notes absolute positions are selected in [0, max_position_embeddings - 1]; Press et al. report that existing methods extrapolate inefficiently and show a 1.3B-parameter language model trained on 1024 tokens extrapolating to 2048 with ALiBi.
Boundary
Transformers over sessions, logs and long text.
Symptom or risk
Learned absolute positions have no entry beyond the trained maximum, and other schemes can degrade quietly beyond training lengths, so long sessions score worse without any error.
Exceptions
Inputs capped below the training length by design.
Trade-offs
Length-stratified evaluation needs labelled long examples that may be rare.
Pitfalls
Averaging the metric across lengths; testing only slightly beyond the maximum.
Lessons and concepts
Claims and their sources
Press, Smith and Lewis (arXiv preprint) report that existing position methods extrapolate to longer inputs inefficiently, and that ALiBi lets a 1.3B-parameter model trained on 1024 tokens extrapolate to 2048.
Hugging Face documents absolute positional embeddings as selected in the range [0, config.max_position_embeddings - 1].