Neural networks, deep learning, NLP · insight-0447
Mask padding in attention, pooling and the loss, then test length invariance
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- e-commerce
Summary: situation, action, reason, boundary
Situation
You batch variable-length texts or event sequences by padding them to the longest item.
Action
Pass a key padding mask to attention, exclude pads from pooling, and set the pad label to the loss's ignore index. Test that one example scores the same alone and padded inside a longer batch.
Reason
Attention and loss functions only skip padding when told which positions are padding.
Boundary
Transformers, RNNs with manual padding, and token-level tagging losses.
Symptom or risk
Unmasked pad tokens receive attention, enter mean pooling and contribute to the loss, so predictions change with the batch's longest item and sequence length becomes a hidden feature.
Exceptions
Packed-sequence utilities for RNNs that handle lengths internally.
Trade-offs
Masks add bookkeeping; dynamic batching by length reduces padding and cost.
Pitfalls
Masking attention but not the mean-pooling step that builds a sentence embedding.
Lessons and concepts
Claims and their sources
PyTorch MultiheadAttention accepts a key_padding_mask of shape (N, S) marking key positions to ignore as padding.
CrossEntropyLoss ignore_index specifies a target value that does not contribute to the input gradient.