Recurrence, vanishing gradients, LSTM and GRU gates, and a complete language model
|
Next Words - Lesson 5 - RNN, LSTM, GRU (pdf)
|
469.84 K |
A learning signal sent back through a plain recurrent network at a factor of 0.8 per step is 0.012 of its size after 20 steps. A sentence of any length is read with one set of weights, so a layer with h = 256 has 65,536 recurrent weights at every length. The gradient to an earlier word is a product of one factor per step. The LSTM has a cell state updated by addition, and its gradient is multiplied by the forget gate alone. The GRU has one update gate and 3 weight blocks where the LSTM has 4, which is 427,776 parameters against 570,368 at h = 256 and d = 300. A complete model of embedding, LSTM and softmax has a perplexity of 138.1 where a trigram has 278.6. Students work through the five notebooks on sequence models, with the text of Pride and Prejudice as data.