Fixed vectors, softmax weights, context vectors, self-attention and GPT-2 small heads
|
Next Words - Lesson 6 - The Attention Revolution (pdf)
|
453.67 K |
A fixed vector of width 256 has 25.6 numbers per word for a 10-word sentence and 5.12 for a 50-word sentence. With attention, the scores 0, 1 and 2 of three source words give the softmax weights 0.090, 0.245 and 0.665, and the context vector (0.335, 1.575) is the weighted sum of the encoder states. Dot products of a query with keys have no parameters. Self-attention over le chat noir has 9 scores, and any two words are one step apart where an RNN has 7 transitions from animal to it. Translation attention, self-attention and image captions over 196 regions are one formula with three slots. In GPT-2 small, 0.63 of the weight of its 144 heads lies on The. Students work through the five notebooks on attention, with the text of Pride and Prejudice as data.