Next Words - Lesson 12 - Scaling Laws

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Next Words - Lesson 12 - Scaling Laws

Power laws, compute-optimal budgets, exact match, mixture of experts, training memory

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • PDF lecture, 21 pages
  • Jupyter notebooks, 4 (zip)
  • Quiz, 5 questions
Tags:
Scaling Laws power law compute budget mixture of experts emergent abilities



What you will learn

  • Compute the factor by which the loss is divided when the parameters grow by a given factor, from the exponent 0.076.
  • Derive the compute-optimal parameters N and tokens D for a budget C at 20 tokens per parameter.
  • Calculate the stored and active parameters of a mixture of experts and the training memory per GPU under ZeRO.

Courselet Content

2 components
Next Words - Lesson 12 - Scaling Laws (pdf)
286.75 K

Requirements

  • Transformers, logarithms, Python

General Overview

Description

In Kaplan's power law with the exponent 0.076, the loss is divided by 1.19 at each tenfold increase in parameters. The cost of a run is C ≈ 6ND FLOPs, and the compute-optimal split is about 20 tokens per parameter. Chinchilla with 70 billion parameters has an MMLU score of 67.6 against 60.0 for Gopher with 280 billion, and GPT-3 is 3.4 times larger than the optimum for its budget. LLaMA-7B is trained on 149 tokens per parameter. At 0.50 per digit, four-digit exact match is 0.0625, so a smooth rise per digit is a late jump. Mixtral has 47 billion stored parameters and 13 billion active parameters per token. The training state of a seven-billion-parameter model is 112 GB, and 14 GB per GPU on eight under ZeRO stage 3. Students work through four notebooks on scaling laws and emergent abilities, with the Pride and Prejudice text as data.

Courses that include this CL

blog
Last Updated 20th November 2026
  • 1
  • Free

Recommended for you

blog
Last Updated 22nd February 2025
  • 1
blog
Last Updated 3rd May 2024
  • 17
blog
Last Updated 10th December 2023
  • 5
blog
Last Updated 19th July 2023
  • 0
  • 0
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 16th January 2023
  • 49
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 24th September 2026
  • 5
blog
Last Updated 19th November 2023
  • 7
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/