Natural Language Processing - Predicting the Next Words

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Natural Language Processing - Predicting the Next Words

Next-word prediction in 18 lessons, from n-gram counts to transformers and calibration

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • 18 courselets in lesson order
  • PDF slides, lectures, primer: 623 pages
  • Jupyter notebooks for 15 lessons (zip)
  • Quizzes, 89 questions
Tags:
Language Models Next-Word Prediction Transformers N-grams NLP



What you will learn

  • Compute n-gram probabilities, perplexity and scaled dot-product attention by hand.
  • Follow one training step of a transformer from the forward pass to the update of the weights.
  • Compare pre-training, prompting, retrieval, alignment and calibration of language models.

Course Content

18 courselets
Next Words - Reading - The Mathematics of Language Models (pdf)
589.43 K
Next Words - Lesson 1 - Predicting the Next Word (pdf)
768.14 K
Next Words - Lesson 2 - Probability, Entropy, Perplexity (pdf)
349.12 K
Next Words - Lesson 3 - N-gram Models, Smoothing (pdf)
345.88 K
Next Words - Lesson 4 - Word Representations (pdf)
389.27 K
Next Words - Lesson 5 - RNN, LSTM, GRU (pdf)
469.75 K
Next Words - Lesson 6 - The Attention Revolution (pdf)
453.84 K
Next Words - Lesson 7 - Sequence-to-Sequence, Decoding (pdf)
437.66 K
The Transformer Architecture (pdf)
2.2 M
14.55 M
Next Words - Lesson 9 - The Transformer in Training (pdf)
459.09 K
Next Words - Lesson 10 - BERT, GPT, T5 (pdf)
385.18 K
Next Words - Lesson 11 - Tokenization, Data at Scale (pdf)
391.35 K
Next Words - Lesson 12 - Scaling Laws (pdf)
286.39 K
Next Words - Lesson 13 - Alignment: RLHF, DPO (pdf)
316.16 K
Next Words - Lesson 14 - In-Context Learning, Prompting (pdf)
298.77 K
Next Words - Lesson 15 - Retrieval, Agents, Multimodal Models (pdf)
306.14 K
Next Words - Lesson 16 - Calibration, Uncertainty (pdf)
389.92 K
Next Words - Lesson 17 - Ethics, Society, the Future (pdf)
271.31 K

Requirements

  • Probability, linear algebra, Python

General Overview

Description

Chapters 1 to 54 of Pride and Prejudice have 107,695 tokens and 6,067 distinct words, so a table of all four-word sequences over these words has about 1.35 × 10^15 cells, and fewer than one cell in ten billion is filled by the novel. A model of the next word is a probability distribution over the vocabulary given the words before it, estimated from far fewer observations than such a table has cells. In the lessons, students move from counted n-grams, smoothing and perplexity through word vectors, recurrent networks and attention to the encoder-decoder transformer of 2017 and one step of its training. The later lessons are on pre-training, tokenization, scaling laws, alignment, prompting, retrieval, calibration and the ethics of language models. The mathematics is in a primer of 29 pages, with every count measured on Pride and Prejudice. Lessons 1 to 17 have a PDF and a quiz, and 15 of them have Jupyter notebooks. Lesson 18 is reserved for project presentations.

Recommended for you

blog
Last Updated 3rd December 2024
  • 5
  • Free
blog
Last Updated 8th March 2025
  • 1
blog
Last Updated 3rd May 2024
  • 17
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 16th January 2023
  • 47
  • Free
blog
Last Updated 16th November 2026
  • 32
  • 3
blog
Last Updated 24th September 2026
  • 5
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/