Fine-tuning, masked and causal losses, perplexity, span corruption, architectures
|
Next Words - Lesson 10 - BERT, GPT, T5 (pdf)
|
385.49 K |
A masked pass over 512 tokens gives 76.8 loss terms and a causal pass gives 512. In fine-tuning, every pre-trained weight is reused and one layer is added on the [CLS] vector. In masked language modelling, 15 per cent of positions are selected, and the probability of sat in the cat [MASK] on the mat is 0.139 under BERT against 0.0007 under GPT-2 after The cat. The causal loss of GPT-2 on The cat sat on the mat is 34.99 bits, a perplexity near 128. In T5, every task is written as text, and span corruption gives 460.8 encoder inputs and 103.4 targets at 512 tokens. At one compute budget the encoder-decoder has a GLUE score of 83.28, and GPT-3 has a TriviaQA score of 71.2 with frozen weights. Students work through four notebooks, from a concepts workbook to a full workbook, on the Pride and Prejudice text.