Neural networks, deep learning, NLP · insight-1465
Guard against rows where every key is masked, because softmax over all minus-infinity is undefined
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A batch contains an empty text or a fully padded sequence, and the attention scores of that row are all filled with minus infinity before the softmax.
Action
Drop or replace empty inputs in the collate function, or always leave one visible token; add a test with an all-pad row that asserts finite outputs on the library version you run.
Reason
PyTorch's reference code fills masked positions with negative infinity before softmax, and a row with no unmasked key has no defined softmax; fused kernels may differ, so test your version.
Boundary
Padded batches, empty strings, retrieval results with no candidates.
Symptom or risk
The softmax becomes 0 divided by 0, giving NaN that spreads through residuals into the loss, but only on the rare batch holding such an input.
Exceptions
Library paths that explicitly document all-masked handling.
Trade-offs
Extra data validation or a placeholder token.
Pitfalls
Replacing NaN with zero in the loss, which hides the broken batch.
Example
Checked locally with NumPy: exponentiating a row of four minus-infinity values after subtracting the row maximum and normalising returns four NaN values.
Lessons and concepts
Claims and their sources
The PyTorch scaled_dot_product_attention reference code fills positions where a boolean mask is False with negative infinity before the softmax.