Recurrent networks, LSTM gates, attention and the transformer as answers to the limits of n-grams
|
The Next Word Problem - Unit 2 (pdf)
|
467.65 K |
An n-gram model has a fixed window of two to five words. In a recurrent network, the whole past is summarised in one hidden vector, updated with the same weights at every position. Gradients are multiplied along the chain of steps and are small for distant words, the vanishing gradient problem. In an LSTM, a separate memory track is updated by addition and controlled by forget, input and output gates. In attention, the words are scored against a query, the scores are passed through a softmax, and the output is a weighted sum of value vectors. In the transformer, recurrence is dropped, self-attention is computed for all positions at once, several attention patterns are computed in parallel as multi-head attention, and word order is added back by sinusoidal positional encoding. N-gram, RNN, LSTM and transformer models are compared by context length, parallelism and memory.