Next-word prediction from n-gram counts and word embeddings to transformers, decoding, fine-tuning and ethics
|
Natural Language Processing - Introduction (pdf)
|
2 M |
Next-word prediction is the training task of language models from bigram counts to transformers. The probability of a word sequence is a product of conditional probabilities by the chain rule, and in an n-gram model the context is limited to the last few words. Unseen n-grams have probability zero without smoothing, and perplexity is the evaluation measure. Students progress from n-gram counts to word embeddings, recurrent networks with gated memory, sequence-to-sequence models with attention, self-attention, BERT and GPT pre-training, byte pair tokenization, decoding, fine-tuning with LoRA, quantization and distillation, and bias measurement.