Neural networks, deep learning, NLP · insight-1468
Read the last real time step of a padded RNN batch, not position minus one
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
An LSTM classifier over right-padded batches uses
output[:, -1]as the sequence summary.Action
Pass the true lengths and pack the batch (or gather the output at
length - 1), and test that one sequence alone and padded with extra steps gives identical logits.Reason
PyTorch's
pack_padded_sequencetakes the real lengths and returns a packed sequence without padding, so the recurrent layer does not step through pad tokens.Boundary
Unidirectional and bidirectional RNNs, LSTMs and GRUs over variable-length input.
Symptom or risk
For shorter sequences the last position is padding, so the summary was updated by pad steps and the same text scores differently depending on the longest sequence in its batch.
Exceptions
Fixed-length inputs.
Trade-offs
Packing adds sorting and a little overhead.
Pitfalls
In a bidirectional layer the reverse direction starts at the pad tokens when not packed; mixing up lengths and the padded size.
Lessons and concepts
Claims and their sources
PyTorch documents pack_padded_sequence as taking the true sequence lengths and returning a PackedSequence without the padding.