The Next Word Problem - Hour 3

  • 0 Rating
  • 0 Reviews
  • 2 Students Enrolled

The Next Word Problem - Hour 3

Pre-training, BERT and GPT, scaling laws, reasoning at answer time and tokenisation

  • 0 Rating
  • 0 Reviews
  • 2 Students Enrolled
  • Free
  • Course Includes
  • PDF slides, 20 pages
  • Room version of the slides (zip)
  • Quiz, 4 questions
Tags:
tokenisation Pre-training BERT GPT Scaling Laws



What you will learn

  • Contrast the bidirectional masking of BERT with the causal masking of GPT.
  • State the power-law form of the scaling laws and the shift from training compute to compute at answer time.
  • Apply byte pair encoding merges to a small corpus and estimate the memory of a training run at about 20 bytes per parame...

Courselet Content

2 components
The Next Word Problem - Unit 3 (pdf)
454.49 K

Requirements

  • The attention and transformer material of Hour 2, basic statistics

General Overview

Description

Before 2018 each task had its own model. Since then a single model is pre-trained on text and adapted many times. BERT is trained to fill masked tokens from both sides, and GPT to predict each token from the tokens to its left. Test loss is a power law in model size, data and compute, with an exponent of about 0.076 for model size, and a compute-optimal recipe of about 20 tokens per parameter. Since 2024, capability is also increased by computing for longer at answer time, in reasoning models. Models are given tokens, subword pieces built by byte pair encoding, with about 1.3 tokens per English word. A single GPT-3 training run is 3.14 x 10^23 floating point operations, 14.8 days on 10,000 V100 GPUs and about 4.6 million dollars. Emergent abilities are hard to predict, and benchmarks are saturated and then replaced.

Courses that include this CL

blog
Last Updated 19th November 2026
  • 2

Recommended for you

blog
Last Updated 8th March 2025
  • 1
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 17th December 2022
  • 30
blog
Last Updated 7th January 2023
  • 22
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 3rd October 2025
  • 97
  • Free
blog
Last Updated 3rd March 2025
  • 86
  • Free
blog
Last Updated 24th September 2026
  • 5
blog
Last Updated 15th January 2026
  • 102
blog
Last Updated 13th December 2022
  • 141
  • Free
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free
blog
Last Updated 6th May 2025
  • 740
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/