Skip to content
ML Academy

Introduced on day 39

Query, key and value

The three linear projections of each token's vector in attention - the query is matched against keys, and the matching weights mix the values.

In detail

With XX of shape (n,dmodel)(n, d_{model}), Q=XWQQ = XW_Q, K=XWKK = XW_K and V=XWVV = XW_V have shape (n,dk)(n, d_k). The roles are not given: they are what training makes the projections do. The weights softmax⁡(QK⊤/dk)\operatorname{softmax}(QK^\top / \sqrt{d_k}) are computed from the input, while WQW_Q, WKW_K, WVW_V are the parameters.