The Next Word Problem - Hour 2

  • 0 Rating
  • 0 Reviews
  • 2 Students Enrolled

The Next Word Problem - Hour 2

Recurrent networks, LSTM gates, attention and the transformer as answers to the limits of n-grams

  • 0 Rating
  • 0 Reviews
  • 2 Students Enrolled
  • Free
  • Course Includes
  • PDF slides, 21 pages
  • Room version of the slides (zip)
  • Quiz, 4 questions
Tags:
Recurrent Neural Networks LSTM Attention Transformers NLP



What you will learn

  • Write the recurrence of an RNN and explain the vanishing gradient problem.
  • Describe the forget, input and output gates of an LSTM and the reduction of the vanishing gradient by an additive cell s...
  • Compute scaled dot-product attention from queries, keys and values, and describe multi-head attention and positional enc...

Courselet Content

2 components
The Next Word Problem - Unit 2 (pdf)
467.65 K

Requirements

  • The n-gram models of Hour 1, vectors, matrices and derivatives

General Overview

Description

An n-gram model has a fixed window of two to five words. In a recurrent network, the whole past is summarised in one hidden vector, updated with the same weights at every position. Gradients are multiplied along the chain of steps and are small for distant words, the vanishing gradient problem. In an LSTM, a separate memory track is updated by addition and controlled by forget, input and output gates. In attention, the words are scored against a query, the scores are passed through a softmax, and the output is a weighted sum of value vectors. In the transformer, recurrence is dropped, self-attention is computed for all positions at once, several attention patterns are computed in parallel as multi-head attention, and word order is added back by sinusoidal positional encoding. N-gram, RNN, LSTM and transformer models are compared by context length, parallelism and memory.

Courses that include this CL

blog
Last Updated 19th November 2026
  • 2

Recommended for you

blog
Last Updated 22nd February 2025
  • 1
blog
Last Updated 10th December 2023
  • 5
blog
Last Updated 30th July 2023
  • 2
blog
Last Updated 19th July 2023
  • 0
  • 0
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 5th June 2025
  • 3
  • Free
blog
Last Updated 16th January 2023
  • 49
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 24th September 2026
  • 5
blog
Last Updated 19th November 2023
  • 7
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/