Power laws, compute-optimal budgets, exact match, mixture of experts, training memory
|
Next Words - Lesson 12 - Scaling Laws (pdf)
|
286.75 K |
In Kaplan's power law with the exponent 0.076, the loss is divided by 1.19 at each tenfold increase in parameters. The cost of a run is C ≈ 6ND FLOPs, and the compute-optimal split is about 20 tokens per parameter. Chinchilla with 70 billion parameters has an MMLU score of 67.6 against 60.0 for Gopher with 280 billion, and GPT-3 is 3.4 times larger than the optimum for its budget. LLaMA-7B is trained on 149 tokens per parameter. At 0.50 per digit, four-digit exact match is 0.0625, so a smooth rise per digit is a late jump. Mixtral has 47 billion stored parameters and 13 billion active parameters per token. The training state of a seven-billion-parameter model is 112 GB, and 14 GB per GPU on eight under ZeRO stage 3. Students work through four notebooks on scaling laws and emergent abilities, with the Pride and Prejudice text as data.