The chain rule, entropy and n-gram counts, from Shannon's 1951 guessing game to GPT-2
|
Next Words - Lesson 1 - Predicting the Next Word (pdf)
|
768.46 K |
A language model is a function from a context to a probability for each of 50,257 tokens, and after the cat sat on the, 17 tokens have half the mass in GPT-2. By the chain rule, assigning a probability to a whole sequence is the same as predicting the next word at every position, and a ten-sentence corpus of 42 tokens is the worked example. Shannon (1951) bounded the entropy of English near 1 bit per character with a guessing game, against 4.75 bits for 27 equally likely symbols. A sentence containing an unseen phrase has probability zero under n-gram counts. N-gram counts are followed by learned word vectors in 2003, recurrence, attention in 2017 and GPT-3 with 175 billion parameters. In the notebooks, students print the full GPT-2 next-token distribution, generate fifty tokens one at a time, and recompute the context-count table for a chosen vocabulary size.