Pre-training, BERT and GPT, scaling laws, reasoning at answer time and tokenisation
|
The Next Word Problem - Unit 3 (pdf)
|
454.49 K |
Before 2018 each task had its own model. Since then a single model is pre-trained on text and adapted many times. BERT is trained to fill masked tokens from both sides, and GPT to predict each token from the tokens to its left. Test loss is a power law in model size, data and compute, with an exponent of about 0.076 for model size, and a compute-optimal recipe of about 20 tokens per parameter. Since 2024, capability is also increased by computing for longer at answer time, in reasoning models. Models are given tokens, subword pieces built by byte pair encoding, with about 1.3 tokens per English word. A single GPT-3 training run is 3.14 x 10^23 floating point operations, 14.8 days on 10,000 V100 GPUs and about 4.6 million dollars. Emergent abilities are hard to predict, and benchmarks are saturated and then replaced.