Next Words - Lesson 6 - The Attention Revolution

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Next Words - Lesson 6 - The Attention Revolution

Fixed vectors, softmax weights, context vectors, self-attention and GPT-2 small heads

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • PDF lecture, 40 pages
  • Jupyter notebooks, 5 (zip)
  • Quiz, 5 questions
Tags:
Attention Self-Attention GPT-2 Encoder-Decoder softmax



What you will learn

  • Compute softmax weights and a context vector from scores and encoder states.
  • Compute dot-product scores of a query against keys and count the scores of self-attention.
  • Fill the three slots of an attention and say what a weight is evidence of.

Courselet Content

2 components
Next Words - Lesson 6 - The Attention Revolution (pdf)
453.67 K

Requirements

  • Vectors, dot product, softmax, Python

General Overview

Description

A fixed vector of width 256 has 25.6 numbers per word for a 10-word sentence and 5.12 for a 50-word sentence. With attention, the scores 0, 1 and 2 of three source words give the softmax weights 0.090, 0.245 and 0.665, and the context vector (0.335, 1.575) is the weighted sum of the encoder states. Dot products of a query with keys have no parameters. Self-attention over le chat noir has 9 scores, and any two words are one step apart where an RNN has 7 transitions from animal to it. Translation attention, self-attention and image captions over 196 regions are one formula with three slots. In GPT-2 small, 0.63 of the weight of its 144 heads lies on The. Students work through the five notebooks on attention, with the text of Pride and Prejudice as data.

Courses that include this CL

blog
Last Updated 20th November 2026
  • 1
  • Free

Recommended for you

blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 8th May 2023
  • 27
blog
Last Updated 17th December 2022
  • 30
blog
Last Updated 16th January 2023
  • 47
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 3rd October 2025
  • 97
  • Free
blog
Last Updated 7th November 2022
  • 157
  • Free
blog
Last Updated 15th January 2026
  • 102
blog
Last Updated 20th August 2025
  • 50
  • Free
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free
blog
Last Updated 20th May 2025
  • 31
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/