Next-word prediction from n-gram counts to transformers, scaling laws, prompting and safety, in four hours
|
The Next Word Problem (pdf)
|
1.13 M |
Predicting the next word of a text from its context is the single task behind language models. The first models are n-gram counts, in which sequences never seen have probability zero, so zeros are removed by smoothing and quality is measured by perplexity. Usable context is widened by word embeddings, recurrent networks and LSTM gates, and the single-vector bottleneck is removed by attention. In the transformer, all positions are processed at once. Current models are built by pre-training with BERT and GPT, scaling, subword tokenisation and reasoning at answer time. Prompting, fine-tuning, retrieval, RLHF, sampling, multimodal models and agents are the applications, together with their safety questions. Worked mathematics is the information content of a word, the six LSTM gate equations, self-attention on three words and the RLHF objective. The 2025 edition is courselet 922, NLP - Predicting the Next Word.