Neural networks, deep learning, NLP · insight-1467
Set padding_idx on embeddings and still mask the pool, or padding length leaks into the features
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A text model uses
nn.Embeddingwith pad id 0 and mean-pools token vectors over the padded length.Action
Create the embedding with
padding_idxso the pad vector starts at zero and gets no gradient, and divide the sum by each sequence's true length through the mask.Reason
PyTorch documents that entries at padding_idx receive no gradient and start as zeros; zeros still count in a mean over padded positions, so the mask remains necessary.
Boundary
Embedding-bag style classifiers, RNN and CNN text models.
Symptom or risk
The pad vector receives gradient and drifts, and the mean divides by the padded length, so the same text gets a different vector depending on which batch it lands in.
Exceptions
Models that already use an attention mask and masked pooling end to end.
Trade-offs
Two small changes.
Pitfalls
Using padding_idx and believing it fixes the mean; setting it to the wrong id after adding special tokens.
Example
A 3-token text padded to 10 positions has a plain mean scaled by 3/10 = 0.3 of the masked mean.
Lessons and concepts
Claims and their sources
PyTorch documents that embedding entries at padding_idx receive no gradient, so that vector is not updated during training, and start as zeros.