NLP - Recurrent Neural Networks

  • 0 Rating
  • 0 Reviews
  • 3 Students Enrolled

NLP - Recurrent Neural Networks

Hidden-state memory, vanishing gradients and the LSTM with forget, input and output gates

  • 0 Rating
  • 0 Reviews
  • 3 Students Enrolled
  • Wishlist
  • Free
  • Course Includes
  • PDF slides, 19 pages
  • LaTeX source and figures (zip)
  • Quiz, 5 questions
Tags:
Recurrent Neural Networks LSTM Vanishing Gradients Sequence Models NLP



What you will learn

  • Describe how the words read so far are held in a hidden state updated with weights shared across time steps
  • Explain the vanishing of gradients over long sequences
  • Implement an LSTM cell with forget, input and output gates and apply gradient clipping

Courselet Content

2 components
NLP - Recurrent Neural Networks (pdf)
309.36 K

Requirements

  • Word embeddings, gradients and the chain rule, basic PyTorch

General Overview

Description

Meaning depends on word order, yet dog bites man and man bites dog have the same set of word vectors. A recurrent network has a hidden state that is updated at every word from the new input and the previous state, with the same weights at every time step. In a plain RNN the gradient is close to zero after 10 to 15 steps of backward propagation, so dependencies beyond that range cannot be learned. In an LSTM the cell state is updated by addition, with forget, input and output gates, and dependencies of 100 or more steps can be learned. Exploding gradients are handled by clipping at a maximum norm of 5, and the gap between training on true words and testing on predicted words is reduced by scheduled sampling.

Courses that include this CL

blog
Last Updated 19th November 2026
  • 5
  • Free

Recommended for you

blog
Last Updated 8th March 2025
  • 1
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 17th December 2022
  • 30
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 20th May 2025
  • 2
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 7th November 2022
  • 11
  • Free
blog
Last Updated 7th November 2022
  • 157
  • Free
blog
Last Updated 3rd March 2025
  • 86
  • Free
blog
Last Updated 24th September 2026
  • 5
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free
blog
Last Updated 20th May 2025
  • 31
  • Free
blog
Last Updated 6th May 2025
  • 740
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/