Introduced on day 39
Query, key and value
The three linear projections of each token's vector in attention - the query is matched against keys, and the matching weights mix the values.
In detail
With of shape , , and have shape . The roles are not given: they are what training makes the projections do. The weights are computed from the input, while , , are the parameters.