Chain rule, maximum likelihood, entropy, KL divergence, perplexity and gradient descent
|
Next Words - Lesson 2 - Probability, Entropy, Perplexity (pdf)
|
349.72 K |
A loss of 3.91 in a training log is a cross-entropy in nats per token, which is a perplexity of 49.9. The probability of a sentence is factored by the chain rule into next-word probabilities. Maximum likelihood is count-and-divide for n-grams, and the product for a sentence is zero when one bigram is unseen. English letter frequencies have an entropy of 4.08 bits per character against a ceiling of 4.75. KL divergence is the excess of cross-entropy over entropy, 0.531 bits forward and 0.737 reverse for one pair of distributions. A perplexity of 15 over subword tokens and one of 20 over words have different denominators, so the two are compared in bits per character. In the notebooks, students compute the entropy of English letter frequencies, plot entropy against temperature, and score a toy sentence under an add-one unigram model.