Six notebooks from n-gram models to BERT and GPT pre-training, each at an easy, an intermediate and an expert level
The notebooks are six next-word predictors in Python, from an n-gram model with smoothing, backoff and interpolation, through a neural language model with word embeddings compared against skip-gram, an RNN and an LSTM with gate analysis, a sequence-to-sequence model with attention evaluated by BLEU and a transformer with self-attention, to the masked and causal pre-training objectives of BERT and GPT. They have an easy start, an intermediate level in which the model is built from scratch and an expert level with the mathematics, from the chain rule and perplexity to LSTM gates, scaled dot-product attention and sinusoidal positional encoding. Interactive widgets are used throughout.