Neural networks, deep learning, NLP · insight-1466

Left-pad batched prompts for decoder-only generation and confirm batched outputs match single runs

Situation
You batch prompts of different lengths for a decoder-only model, using the tokenizer's default right padding.
Action
Set the tokenizer's padding side to left, define a pad token if missing, pass the attention mask, and assert that greedy batched outputs match single-prompt outputs on a sample (up to rare numerical tie-breaks).
Reason
Hugging Face documents that LLMs are not trained to continue generation from padding tokens, so the padding side must be left for batched generation, and shows a right-padded example producing a degenerate continuation.
Boundary
Batched inference with decoder-only models.

Symptom or risk

The model has to continue from pad tokens for the shorter prompts, giving degenerate or truncated text that differs from running each prompt alone.

Encoder models and classification heads, which normally use right padding with a mask.

None beyond a configuration check.

Setting the pad token to the end-of-sequence token and then masking real end tokens; slicing off the prompt with the wrong length.

Hugging Face documents that LLMs are not trained to continue generation from padding tokens, so padding_side needs to be left for batched generation.
Text generation, Pitfalls: padding side (Hugging Face Transformers documentation)