Scaling laws, emergent abilities, sparse and flash attention, few-shot prompting and parallel training
|
NLP - Advanced Transformers (pdf)
|
455.89 K |
Self-attention has time and memory cost O(n^2) in the sequence length n. Sparse attention with local and global patterns has cost O(n sqrt n), FlashAttention is an IO-aware algorithm for exact attention that is 2 to 4 times faster, and rotary position embeddings have better length extrapolation. Some abilities are absent in small models and present in large ones, and scaling laws give the loss as a function of parameters, data and compute. In few-shot prompting, a task is specified by examples in the prompt, and accuracy depends on the wording of the prompt. Training at scale is organized with model parallelism across GPUs, pipeline parallelism over micro-batches and mixed precision with FP16 and FP32 master weights.