Next Words - Lesson 1 - Predicting the Next Word

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Next Words - Lesson 1 - Predicting the Next Word

The chain rule, entropy and n-gram counts, from Shannon's 1951 guessing game to GPT-2

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • PDF slides, 46 pages
  • Jupyter notebooks, 2 (zip)
  • Quiz, 5 questions
Tags:
chain rule entropy Next-Word Prediction N-grams GPT-2



What you will learn

  • Write the probability of a word sequence as a product of next-word probabilities with the chain rule.
  • Compute entropy in bits, including 1.58 bits for three equally likely words and 4.75 bits for 27 equal symbols.
  • State why an unseen phrase has probability zero in a trigram model and why dog and puppy are unrelated entries in a coun...

Courselet Content

2 components
Next Words - Lesson 1 - Predicting the Next Word (pdf)
768.46 K

Requirements

  • Probability basics, vectors, Python

General Overview

Description

A language model is a function from a context to a probability for each of 50,257 tokens, and after the cat sat on the, 17 tokens have half the mass in GPT-2. By the chain rule, assigning a probability to a whole sequence is the same as predicting the next word at every position, and a ten-sentence corpus of 42 tokens is the worked example. Shannon (1951) bounded the entropy of English near 1 bit per character with a guessing game, against 4.75 bits for 27 equally likely symbols. A sentence containing an unseen phrase has probability zero under n-gram counts. N-gram counts are followed by learned word vectors in 2003, recurrence, attention in 2017 and GPT-3 with 175 billion parameters. In the notebooks, students print the full GPT-2 next-token distribution, generate fifty tokens one at a time, and recompute the context-count table for a chosen vocabulary size.

Courses that include this CL

blog
Last Updated 20th November 2026
  • 1
  • Free

Recommended for you

blog
Last Updated 8th March 2025
  • 1
blog
Last Updated 27th May 2024
  • 2
blog
Last Updated 25th July 2023
  • 2
blog
Last Updated 19th July 2023
  • 0
  • 0
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 16th January 2023
  • 49
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 24th September 2026
  • 5
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/