Bigram and trigram models, add-one smoothing and perplexity for next-word prediction
|
Foundations and Statistical Language Models (pdf)
|
376.83 K |
The probability of a word sequence is the product of P(wi | w1, ..., wi-1). In an n-gram model the context is limited to the last n-1 words, so that P(cat | the) equals count(the cat) divided by count(the). Bigram probabilities are worked out by hand on a three-sentence corpus and then implemented in Python with START and END markers. A word never seen in training gives the whole sentence probability zero, and probability mass is reserved for unseen events by add-one smoothing and its refinements. Perplexity is the measure of how surprised a model is by text, and lower is better. N-gram models have no representation of long-distance dependencies, word meaning or context switching.