Next Words - Reading - The Mathematics of Language Models

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Next Words - Reading - The Mathematics of Language Models

Counting, smoothing, entropy, word vectors, attention and decoding on one novel

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • PDF primer, 29 pages
  • Quiz, 5 questions
Tags:
Language Models N-grams entropy Attention Decoding



What you will learn

  • Compute the probability and the bits of a sentence from bigram counts with the chain rule.
  • Derive cross-entropy as the sum of entropy and KL divergence and read perplexity as an effective number of choices.
  • Explain why attention scores are divided by the square root of the head width and how a next-word distribution changes w...

Courselet Content

1 components
Next Words - Reading - The Mathematics of Language Models (pdf)
587.7 K

Requirements

  • Probability, logarithms, basic calculus

General Overview

Description

After the words "he had", the training text of a novel contains 157 continuations over 86 different words, and the most probable of them has 15% of the mass. By the chain rule the probability of a sentence is a product of such distributions and its cost is a sum of bits. A bigram table filled by counting is 99.87% empty, and the zeros are repaired by add-k, Good-Turing and interpolation. The repairs are ranked by entropy, cross-entropy and perplexity in bits per character, from 2.520 for a uniform guess to 1.623 for an interpolated trigram. A network on word vectors has about 592 thousand parameters and a window of 3 words, and attention with a causal mask is a weighted average over every earlier position. Decoding is a separate decision, since greedy output has a cost of 3.04 bits a word against 6.98 for real prose.

Courses that include this CL

blog
Last Updated 20th November 2026
  • 1
  • Free

Recommended for you

blog
Last Updated 15th January 2025
  • 9
blog
Last Updated 10th December 2023
  • 5
blog
Last Updated 19th November 2026
  • 8
  • Free
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 27th February 2026
  • 13
  • Free
blog
Last Updated 16th January 2023
  • 49
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 31st July 2025
  • 32
blog
Last Updated 20th August 2025
  • 50
  • Free
blog
Last Updated 19th November 2023
  • 7
blog
Last Updated 19th November 2026
  • 7
  • Free
blog
Last Updated 13th December 2022
  • 141
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/