Next Words - Lesson 9 - The Transformer in Training

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Next Words - Lesson 9 - The Transformer in Training

Known data, loss, gradient step, learning rate and decoder-only training

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • PDF slides, 32 pages
  • Jupyter notebooks, 6 (zip)
  • Further reading PDF, 48 pages (zip)
  • Quiz, 5 questions
Tags:
Transformer cross-entropy training learning rate decoder-only



What you will learn

  • Compute the shapes of the vectors, the attention weights and the scores along the path of one sentence pair through the...
  • Derive the gradient p minus q of the cross-entropy at the scores, with and without label smoothing.
  • Evaluate the warm-up learning-rate schedule and the next-token loss of a decoder-only transformer under a causal mask.

Courselet Content

2 components
Next Words - Lesson 9 - The Transformer in Training (pdf)
459.57 K

Requirements

  • Attention, softmax, cross-entropy, Python

General Overview

Description

The 2017 Transformer has 65 million weights that are random at step 0 and are fitted to 4.5 million known sentence pairs. The path of one pair through the encoder and the decoder is followed on a miniature model with width 16 and a vocabulary of 9 tokens, for the cat runs and die Katze läuft. The shapes along the path are written in the symbols n, m, d, h and V. The loss is the mean cross-entropy with label smoothing 0.1, and a training step is a forward pass, a loss, a gradient equal to p minus q and an Adam update. The learning rate grows over 4,000 warm-up steps to 7.0 · 10⁻⁴. In a decoder-only transformer, one sequence shifted by one token is input and target, under a causal mask. Students work through six notebooks, with the Pride and Prejudice text as data.

Courses that include this CL

blog
Last Updated 20th November 2026
  • 1
  • Free

Recommended for you

blog
Last Updated 3rd December 2024
  • 5
  • Free
blog
Last Updated 2nd August 2023
  • 2
  • Free
blog
Last Updated 30th July 2023
  • 2
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 17th December 2022
  • 30
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 3rd October 2025
  • 97
  • Free
blog
Last Updated 3rd March 2025
  • 86
  • Free
blog
Last Updated 15th January 2026
  • 102
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free
blog
Last Updated 6th May 2025
  • 740
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/