NLP - Predicting the Next Word

  • 0 Rating
  • 0 Reviews
  • 2 Students Enrolled

NLP - Predicting the Next Word

The next-word task from n-gram counts through attention and pre-training to decoding, fine-tuning and RLHF

  • 0 Rating
  • 0 Reviews
  • 2 Students Enrolled
  • Wishlist
  • Free
  • Course Includes
  • PDF slides, 53 pages
  • Quiz, 4 questions
Tags:
NLP Next-Word Prediction Language Models Transformers Perplexity



What you will learn

  • Compute the information content of a word as minus log2 of its probability given the context, and the entropy of a predi...
  • Compare n-grams, RNNs, sequence-to-sequence models and the transformer as solutions of one prediction task
  • Describe pre-training with BERT and GPT, scaling laws, byte pair tokenization, decoding and the RLHF objective

Courselet Content

1 components
NLP - Predicting the next word (pdf)
703.25 K

Requirements

  • Probability and logarithms, basic Python

General Overview

Description

Next-word prediction is the task behind n-gram models, recurrent networks, BERT, GPT and the transformer. The information content of a word w after context c is -log2 P(w | c), entropy is the average information, and perplexity is the measure of prediction quality. Counting with maximum likelihood estimates and add-one smoothing is the first solution, followed by RNNs, sequence-to-sequence models and scaled dot-product attention, with a causal mask in the full transformer block. Students then follow pre-training with masked language modeling and GPT, scaling laws, byte pair tokenization, fine-tuning and prompting as two ways to adapt a model, decoding from probabilities, and the RLHF objective of maximizing reward under a KL penalty to a reference model. The 2026 edition is courselet 100099, The Next Word Problem.

Courses that include this CL

blog
Last Updated 19th November 2026
  • 5
  • Free

Recommended for you

blog
Last Updated 3rd December 2024
  • 5
  • Free
blog
Last Updated 8th March 2025
  • 1
blog
Last Updated 19th July 2023
  • 0
  • 0
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 8th May 2023
  • 27
blog
Last Updated 16th January 2023
  • 47
  • Free
blog
Last Updated 16th January 2023
  • 49
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 24th September 2026
  • 5
blog
Last Updated 20th August 2025
  • 50
  • Free
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/