The Next Word Problem

  • 0 Rating
  • 0 Reviews
  • 4 Students Enrolled

The Next Word Problem

Next-word prediction from n-gram counts to transformers, scaling laws, prompting and safety, in four hours

  • 0 Rating
  • 0 Reviews
  • 4 Students Enrolled
  • Free
  • Course Includes
  • PDF slides, 93 pages
  • Quiz, 4 questions
Tags:
Next-Word Prediction Language Models Transformers N-grams NLP



What you will learn

  • Compute n-gram probabilities with smoothing and evaluate a language model by perplexity.
  • Trace the path from recurrent networks and LSTM gates to scaled dot-product attention and the transformer, including a t...
  • Compare pre-training, prompting, retrieval, fine-tuning and RLHF, and sampling by temperature, top-k and nucleus.

Courselet Content

1 components
The Next Word Problem (pdf)
1.13 M

Requirements

  • Probability, linear algebra, Python

General Overview

Description

Predicting the next word of a text from its context is the single task behind language models. The first models are n-gram counts, in which sequences never seen have probability zero, so zeros are removed by smoothing and quality is measured by perplexity. Usable context is widened by word embeddings, recurrent networks and LSTM gates, and the single-vector bottleneck is removed by attention. In the transformer, all positions are processed at once. Current models are built by pre-training with BERT and GPT, scaling, subword tokenisation and reasoning at answer time. Prompting, fine-tuning, retrieval, RLHF, sampling, multimodal models and agents are the applications, together with their safety questions. Worked mathematics is the information content of a word, the six LSTM gate equations, self-attention on three words and the RLHF objective. The 2025 edition is courselet 922, NLP - Predicting the Next Word.

Courses that include this CL

blog
Last Updated 19th November 2026
  • 2

Recommended for you

blog
Last Updated 15th January 2025
  • 9
blog
Last Updated 3rd May 2024
  • 17
blog
Last Updated 10th December 2023
  • 5
blog
Last Updated 19th November 2026
  • 8
  • Free
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 16th January 2023
  • 48
  • Free
blog
Last Updated 16th January 2023
  • 47
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 15th October 2026
  • 53
  • Free
blog
Last Updated 20th August 2025
  • 50
  • Free
blog
Last Updated 19th November 2023
  • 7
blog
Last Updated 19th November 2026
  • 7
  • Free
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/