N-gram language models, smoothing and perplexity, with a counting lab
|
The Next Word Problem - Unit 1 (pdf)
|
365.91 K |
The next word of a text is described by a conditional probability given the words before it, and the exact form for whole sentences is the chain rule of probability. The exact form is unusable because long contexts are almost never seen twice, so in n-gram models only the last n-1 words are kept. Probabilities are estimated by counting and dividing. Word sequences never seen have probability zero, which is corrected by smoothing, and perplexity, equal to 2 to the power of the entropy, is the effective number of equally likely next words. On a toy corpus, the probability of mat after "The cat sat on the" is 2.3 percent for a unigram model, 8.5 percent for a bigram model and 61.5 percent for a 4-gram model. Students reproduce the example in twenty lines of Python and run a counting lab in a notebook.