Neural networks, deep learning, NLP · insight-1461

Check mask polarity at every library boundary, because True can mean attend or ignore

Situation
Custom transformer code combines a Hugging Face attention_mask with nn.MultiheadAttention or scaled_dot_product_attention.
Action
Convert once with a named helper, assert the convention at the boundary, and test that a short sequence gives the same output alone and padded inside a batch.
Reason
Hugging Face uses 1 for tokens to attend and 0 for padding; MultiheadAttention's boolean key_padding_mask marks positions to ignore with True, whereas scaled_dot_product_attention's boolean mask uses True for positions that take part.
Boundary
Any code mixing masks from tokenizers, nn.MultiheadAttention and fused attention functions.

Symptom or risk

An inverted mask attends to padding and ignores real tokens; shapes match, the loss still falls, and the model is just mediocre.

Using one library end to end with its own mask helper.

One extra unit test and helper.

Fixing the symptom by inverting in one place and double-inverting elsewhere; float masks that are added, not selected.

In Hugging Face tokenizers the attention mask has 1 for tokens that should be attended to and 0 for padded values.
Glossary (Hugging Face Transformers documentation)
PyTorch MultiheadAttention treats True in key_padding_mask as a position to ignore.
MultiheadAttention (PyTorch documentation)
PyTorch scaled_dot_product_attention treats True in a boolean attn_mask as a position that takes part in attention, the inverse of MultiheadAttention's key_padding_mask convention.
torch.nn.functional.scaled_dot_product_attention (PyTorch documentation)