Known data, loss, gradient step, learning rate and decoder-only training
|
Next Words - Lesson 9 - The Transformer in Training (pdf)
|
459.57 K |
The 2017 Transformer has 65 million weights that are random at step 0 and are fitted to 4.5 million known sentence pairs. The path of one pair through the encoder and the decoder is followed on a miniature model with width 16 and a vocabulary of 9 tokens, for the cat runs and die Katze läuft. The shapes along the path are written in the symbols n, m, d, h and V. The loss is the mean cross-entropy with label smoothing 0.1, and a training step is a forward pass, a loss, a gradient equal to p minus q and an Adam update. The learning rate grows over 4,000 warm-up steps to 7.0 · 10⁻⁴. In a decoder-only transformer, one sequence shifted by one token is input and target, under a causal mask. Students work through six notebooks, with the Pride and Prejudice text as data.