Skip to content
ML Academy

Introduced on day 39

Positional encoding

Information about token position added to or combined with the input vectors, because self-attention alone is blind to order.

In detail

The original transformer added fixed sinusoids of different frequencies; many models learn position vectors or use relative or rotary schemes. Without any, "refund then delayed" and "delayed then refund" produce the same set of outputs (permuted).