Next Words - Lesson 10 - BERT, GPT, T5

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Next Words - Lesson 10 - BERT, GPT, T5

Fine-tuning, masked and causal losses, perplexity, span corruption, architectures

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • PDF lecture, 39 pages
  • Jupyter notebooks, 4 (zip)
  • Quiz, 5 questions
Tags:
BERT GPT Pre-training Perplexity T5



What you will learn

  • Compute the number of loss terms of a masked pass and of a causal pass over a text of T tokens.
  • Convert a loss in bits per token into a perplexity and into a number of equally likely words.
  • Derive the lengths of the encoder input and the decoder target in T5 span corruption.

Courselet Content

2 components
Next Words - Lesson 10 - BERT, GPT, T5 (pdf)
385.49 K

Requirements

  • Softmax, cross-entropy, transformer basics

General Overview

Description

A masked pass over 512 tokens gives 76.8 loss terms and a causal pass gives 512. In fine-tuning, every pre-trained weight is reused and one layer is added on the [CLS] vector. In masked language modelling, 15 per cent of positions are selected, and the probability of sat in the cat [MASK] on the mat is 0.139 under BERT against 0.0007 under GPT-2 after The cat. The causal loss of GPT-2 on The cat sat on the mat is 34.99 bits, a perplexity near 128. In T5, every task is written as text, and span corruption gives 460.8 encoder inputs and 103.4 targets at 512 tokens. At one compute budget the encoder-decoder has a GLUE score of 83.28, and GPT-3 has a TriviaQA score of 71.2 with frozen weights. Students work through four notebooks, from a concepts workbook to a full workbook, on the Pride and Prejudice text.

Courses that include this CL

blog
Last Updated 20th November 2026
  • 1
  • Free

Recommended for you

blog
Last Updated 10th December 2023
  • 5
blog
Last Updated 30th July 2023
  • 2
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 16th January 2023
  • 48
  • Free
blog
Last Updated 16th January 2023
  • 47
  • Free
blog
Last Updated 11th September 2024
  • 123
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 7th November 2022
  • 11
  • Free
blog
Last Updated 19th November 2023
  • 7
blog
Last Updated 21st March 2025
  • 251
  • Free
blog
Last Updated 6th May 2025
  • 740
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/