Hidden-state memory, vanishing gradients and the LSTM with forget, input and output gates
|
NLP - Recurrent Neural Networks (pdf)
|
309.36 K |
Meaning depends on word order, yet dog bites man and man bites dog have the same set of word vectors. A recurrent network has a hidden state that is updated at every word from the new input and the previous state, with the same weights at every time step. In a plain RNN the gradient is close to zero after 10 to 15 steps of backward propagation, so dependencies beyond that range cannot be learned. In an LSTM the cell state is updated by addition, with forget, input and output gates, and dependencies of 100 or more steps can be learned. Exploding gradients are handled by clipping at a maximum norm of 5, and the gap between training on true words and testing on predicted words is reduced by scheduled sampling.