Introduced on day 39
Self-attention
Attention in which queries, keys and values all come from the same sequence, so every position can read every other position in one step.
In detail
It has no built-in notion of order (permuting the input permutes the output), so positional information is added. Cost grows with in the score matrix. Multi-head attention runs several such maps in parallel with separate projections and mixes them with an output matrix.