Next-word prediction in 18 lessons, from n-gram counts to transformers and calibration
|
Next Words - Reading - The Mathematics of Language Models (pdf)
|
589.43 K |
|
Next Words - Lesson 1 - Predicting the Next Word (pdf)
|
768.14 K |
|
Next Words - Lesson 2 - Probability, Entropy, Perplexity (pdf)
|
349.12 K |
|
Next Words - Lesson 3 - N-gram Models, Smoothing (pdf)
|
345.88 K |
|
Next Words - Lesson 4 - Word Representations (pdf)
|
389.27 K |
|
Next Words - Lesson 5 - RNN, LSTM, GRU (pdf)
|
469.75 K |
|
Next Words - Lesson 6 - The Attention Revolution (pdf)
|
453.84 K |
|
Next Words - Lesson 7 - Sequence-to-Sequence, Decoding (pdf)
|
437.66 K |
|
The Transformer Architecture (pdf)
|
2.2 M | |
|
|
14.55 M |
|
Next Words - Lesson 9 - The Transformer in Training (pdf)
|
459.09 K |
|
Next Words - Lesson 10 - BERT, GPT, T5 (pdf)
|
385.18 K |
|
Next Words - Lesson 11 - Tokenization, Data at Scale (pdf)
|
391.35 K |
|
Next Words - Lesson 12 - Scaling Laws (pdf)
|
286.39 K |
|
Next Words - Lesson 13 - Alignment: RLHF, DPO (pdf)
|
316.16 K |
|
Next Words - Lesson 14 - In-Context Learning, Prompting (pdf)
|
298.77 K |
|
Next Words - Lesson 15 - Retrieval, Agents, Multimodal Models (pdf)
|
306.14 K |
|
Next Words - Lesson 16 - Calibration, Uncertainty (pdf)
|
389.92 K |
|
Next Words - Lesson 17 - Ethics, Society, the Future (pdf)
|
271.31 K |
Chapters 1 to 54 of Pride and Prejudice have 107,695 tokens and 6,067 distinct words, so a table of all four-word sequences over these words has about 1.35 × 10^15 cells, and fewer than one cell in ten billion is filled by the novel. A model of the next word is a probability distribution over the vocabulary given the words before it, estimated from far fewer observations than such a table has cells. In the lessons, students move from counted n-grams, smoothing and perplexity through word vectors, recurrent networks and attention to the encoder-decoder transformer of 2017 and one step of its training. The later lessons are on pre-training, tokenization, scaling laws, alignment, prompting, retrieval, calibration and the ethics of language models. The mathematics is in a primer of 29 pages, with every count measured on Pride and Prejudice. Lessons 1 to 17 have a PDF and a quiz, and 15 of them have Jupyter notebooks. Lesson 18 is reserved for project presentations.