Next Words - Lesson 5 - RNN, LSTM, GRU

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Next Words - Lesson 5 - RNN, LSTM, GRU

Recurrence, vanishing gradients, LSTM and GRU gates, and a complete language model

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • PDF lecture, 37 pages
  • Jupyter notebooks, 5 (zip)
  • Quiz, 5 questions
Tags:
LSTM Vanishing Gradients Perplexity RNN GRU



What you will learn

  • Compute the share of a gradient that is left after k steps at a given factor per step.
  • Update an LSTM cell entry and a GRU entry by hand from given gate values.
  • Count the parameters of an RNN, an LSTM and a GRU from the sizes h and d.

Courselet Content

2 components
Next Words - Lesson 5 - RNN, LSTM, GRU (pdf)
469.84 K

Requirements

  • Matrix products, derivatives, softmax, Python

General Overview

Description

A learning signal sent back through a plain recurrent network at a factor of 0.8 per step is 0.012 of its size after 20 steps. A sentence of any length is read with one set of weights, so a layer with h = 256 has 65,536 recurrent weights at every length. The gradient to an earlier word is a product of one factor per step. The LSTM has a cell state updated by addition, and its gradient is multiplied by the forget gate alone. The GRU has one update gate and 3 weight blocks where the LSTM has 4, which is 427,776 parameters against 570,368 at h = 256 and d = 300. A complete model of embedding, LSTM and softmax has a perplexity of 138.1 where a trigram has 278.6. Students work through the five notebooks on sequence models, with the text of Pride and Prejudice as data.

Courses that include this CL

blog
Last Updated 20th November 2026
  • 1
  • Free

Recommended for you

blog
Last Updated 3rd December 2024
  • 5
  • Free
blog
Last Updated 3rd May 2024
  • 17
blog
Last Updated 10th December 2023
  • 5
blog
Last Updated 14th November 2023
  • 0
  • 0
blog
Last Updated 19th November 2026
  • 8
  • Free
blog
Last Updated 19th July 2023
  • 0
  • 0
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 16th January 2023
  • 49
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 19th November 2023
  • 7
blog
Last Updated 19th November 2026
  • 7
  • Free
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/