Self-attention with queries, keys and values, multi-head attention, positional encoding and the transformer block
|
NLP - The Transformer Revolution (pdf)
|
395.77 K |
Attention(Q, K, V) equals softmax(QK^T / sqrt(dk)) V, and in self-attention the queries, keys and values all come from the same sequence. Self-attention has no notion of position, so sinusoidal positional encodings are added. Multi-head attention has 8 heads of dimension 64 for a model dimension of 512. A transformer block has self-attention and a feed-forward layer of dimension 2048, each with a residual connection. Because all positions are computed in parallel, training is faster than with RNNs. Encoder-only models such as BERT and decoder-only models such as GPT belong to the transformer family tree. The encoder-decoder transformer of 2017 is treated in detail in courselet 100147, The Transformer Architecture.