Next Words - Lesson 2 - Probability, Entropy, Perplexity

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Next Words - Lesson 2 - Probability, Entropy, Perplexity

Chain rule, maximum likelihood, entropy, KL divergence, perplexity and gradient descent

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • PDF slides, 42 pages
  • Jupyter notebooks, 2 (zip)
  • Quiz, 5 questions
Tags:
entropy Perplexity probability cross-entropy maximum likelihood



What you will learn

  • Decompose the probability of a sentence with the chain rule and estimate each factor by maximum likelihood.
  • Compute entropy, cross-entropy and KL divergence in bits, such as 0.531 bits for P = (0.9, 0.1) against Q = (0.5, 0.5).
  • Convert a loss of 3.91 nats into bits and a perplexity of 49.9, and state when two perplexities are comparable.

Courselet Content

2 components
Next Words - Lesson 2 - Probability, Entropy, Perplexity (pdf)
349.72 K

Requirements

  • Conditional probability, logarithms, Python

General Overview

Description

A loss of 3.91 in a training log is a cross-entropy in nats per token, which is a perplexity of 49.9. The probability of a sentence is factored by the chain rule into next-word probabilities. Maximum likelihood is count-and-divide for n-grams, and the product for a sentence is zero when one bigram is unseen. English letter frequencies have an entropy of 4.08 bits per character against a ceiling of 4.75. KL divergence is the excess of cross-entropy over entropy, 0.531 bits forward and 0.737 reverse for one pair of distributions. A perplexity of 15 over subword tokens and one of 20 over words have different denominators, so the two are compared in bits per character. In the notebooks, students compute the entropy of English letter frequencies, plot entropy against temperature, and score a toy sentence under an add-one unigram model.

Courses that include this CL

blog
Last Updated 20th November 2026
  • 1
  • Free

Recommended for you

blog
Last Updated 8th March 2025
  • 1
blog
Last Updated 10th December 2023
  • 5
blog
Last Updated 14th November 2023
  • 0
  • 0
blog
Last Updated 30th July 2023
  • 2
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 16th January 2023
  • 47
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 20th May 2025
  • 2
blog
Last Updated 7th November 2022
  • 88
  • Free
blog
Last Updated 24th September 2026
  • 5
blog
Last Updated 19th November 2023
  • 7
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/