Neural networks, deep learning, NLP · insight-1470
Use a 1-D CNN with max-pooling as a fast pattern-detector baseline for long sequences
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- energy, e-commerce
Summary: situation, action, reason, boundary
Situation
A label depends on short local patterns in long event or text sequences, and recurrent models are slow to train.
Action
Train a 1-D convolution with kernel widths of 2 to 5 and global max-pooling over time, compare with TF-IDF n-grams and a GRU, and read the widths as the longest pattern each filter can see.
Reason
Kim reported that a simple CNN over pretrained word vectors with little tuning did well on sentence classification benchmarks, beating prior results on four of seven tasks.
Boundary
Text, click sequences, sensor windows.
Symptom or risk
Time goes into a recurrent or attention model while a convolutional detector could have shown whether local patterns already explain the label.
Exceptions
Dependencies far beyond kernel width, which need dilation, stacking or attention.
Trade-offs
Global max-pooling discards where a pattern occurred.
Pitfalls
Letting padded positions win the max; widths chosen from the image-CNN habit of 3.
Lessons and concepts
Claims and their sources
Kim reports that a simple CNN over pretrained word vectors with little hyperparameter tuning performs well on sentence classification and beats prior best results on four of seven tasks.