Next Words - Lesson 3 - N-gram Models, Smoothing

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Next Words - Lesson 3 - N-gram Models, Smoothing

N-gram models on a toy corpus with add-k, Good-Turing, Kneser-Ney and perplexity

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • PDF slides, 44 pages
  • Jupyter notebooks, 2 (zip)
  • Quiz, 5 questions
Tags:
N-grams Smoothing Perplexity Language Models sparsity



What you will learn

  • Build a bigram model from counts on a four-sentence corpus and check that every row sums to one.
  • Apply add-k, Good-Turing, Kneser-Ney and interpolation to unseen n-grams and compare their trade-offs.
  • Compute a perplexity from a total log probability and state when two perplexities are comparable.

Courselet Content

2 components
Next Words - Lesson 3 - N-gram Models, Smoothing (pdf)
345.28 K

Requirements

  • Conditional probability, entropy, Python

General Overview

Description

A 5-gram model with modified Kneser-Ney smoothing has a perplexity of about 141 on a standard 10,000-word vocabulary. On a toy corpus of four sentences with 16 distinct bigrams, the probability of cat after the is 3/8 = 0.375, and a sentence with one unseen bigram has probability zero. Over 99 percent of a bigram table is unobserved. The repairs are add-k, Good-Turing, Kneser-Ney and interpolation, and under add-one smoothing the probability of a bigram seen 10 times falls from 0.10 to 0.00022. The gain from a higher order is smaller at each step, 6.3 percent from trigram to 4-gram on 14 million words, and a subject fourteen words back lies outside a 5-gram window. In the notebooks, students build the bigram table, generate text from it, and score the rug sat on the cat under maximum likelihood, add-one and Kneser-Ney.

Courses that include this CL

blog
Last Updated 20th November 2026
  • 1
  • Free

Recommended for you

blog
Last Updated 3rd December 2024
  • 5
  • Free
blog
Last Updated 3rd May 2024
  • 17
blog
Last Updated 30th July 2023
  • 2
blog
Last Updated 19th July 2023
  • 0
  • 0
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 16th January 2023
  • 47
  • Free
blog
Last Updated 16th January 2023
  • 49
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/