NLP - Advanced Transformers

  • 0 Rating
  • 0 Reviews
  • 2 Students Enrolled

NLP - Advanced Transformers

Scaling laws, emergent abilities, sparse and flash attention, few-shot prompting and parallel training

  • 0 Rating
  • 0 Reviews
  • 2 Students Enrolled
  • Wishlist
  • Free
  • Course Includes
  • PDF slides, 20 pages
  • LaTeX source and figures (zip)
  • Quiz, 4 questions
Tags:
Scaling Laws Sparse Attention Few-Shot Learning NLP Large Language Models



What you will learn

  • Describe emergent abilities and the scaling laws of large language models
  • Implement sparse attention with local and global patterns and compare its cost with dense attention
  • Write few-shot prompts and name the methods for training at scale, which are model parallelism, pipeline parallelism and...

Courselet Content

2 components
NLP - Advanced Transformers (pdf)
455.89 K

Requirements

  • Transformer architecture, BERT and GPT, basic PyTorch

General Overview

Description

Self-attention has time and memory cost O(n^2) in the sequence length n. Sparse attention with local and global patterns has cost O(n sqrt n), FlashAttention is an IO-aware algorithm for exact attention that is 2 to 4 times faster, and rotary position embeddings have better length extrapolation. Some abilities are absent in small models and present in large ones, and scaling laws give the loss as a function of parameters, data and compute. In few-shot prompting, a task is specified by examples in the prompt, and accuracy depends on the wording of the prompt. Training at scale is organized with model parallelism across GPUs, pipeline parallelism over micro-batches and mixed precision with FP16 and FP32 master weights.

Courses that include this CL

blog
Last Updated 19th November 2026
  • 5
  • Free

Recommended for you

blog
Last Updated 3rd December 2024
  • 5
  • Free
blog
Last Updated 2nd August 2023
  • 2
  • Free
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 17th December 2022
  • 30
blog
Last Updated 16th January 2023
  • 47
  • Free
blog
Last Updated 7th January 2023
  • 22
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 13th December 2022
  • 141
  • Free
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/