The Next Word Problem - Hour 1

  • 0 Rating
  • 0 Reviews
  • 3 Students Enrolled

The Next Word Problem - Hour 1

N-gram language models, smoothing and perplexity, with a counting lab

  • 0 Rating
  • 0 Reviews
  • 3 Students Enrolled
  • Free
  • Course Includes
  • PDF slides, 25 pages
  • Room version of the slides (zip)
  • Quiz, 4 questions
Tags:
N-grams Language Models Perplexity Smoothing NLP



What you will learn

  • State next-word prediction formally and derive the chain rule of probability.
  • Compute bigram and trigram probabilities by counting, and explain why smoothing is needed under data sparsity.
  • Evaluate a language model by perplexity, equal to 2 to the power of the entropy.

Courselet Content

2 components
The Next Word Problem - Unit 1 (pdf)
365.91 K

Requirements

  • Conditional probability, Python

General Overview

Description

The next word of a text is described by a conditional probability given the words before it, and the exact form for whole sentences is the chain rule of probability. The exact form is unusable because long contexts are almost never seen twice, so in n-gram models only the last n-1 words are kept. Probabilities are estimated by counting and dividing. Word sequences never seen have probability zero, which is corrected by smoothing, and perplexity, equal to 2 to the power of the entropy, is the effective number of equally likely next words. On a toy corpus, the probability of mat after "The cat sat on the" is 2.3 percent for a unigram model, 8.5 percent for a bigram model and 61.5 percent for a 4-gram model. Students reproduce the example in twenty lines of Python and run a counting lab in a notebook.

Courses that include this CL

blog
Last Updated 19th November 2026
  • 2

Recommended for you

blog
Last Updated 3rd May 2024
  • 17
blog
Last Updated 10th December 2023
  • 5
blog
Last Updated 14th November 2023
  • 0
  • 0
blog
Last Updated 19th November 2026
  • 8
  • Free
blog
Last Updated 19th July 2023
  • 0
  • 0
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 19th November 2023
  • 7
blog
Last Updated 19th November 2026
  • 7
  • Free
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/