NLP - The Transformer Revolution

  • 0 Rating
  • 0 Reviews
  • 4 Students Enrolled

NLP - The Transformer Revolution

Self-attention with queries, keys and values, multi-head attention, positional encoding and the transformer block

  • 0 Rating
  • 0 Reviews
  • 4 Students Enrolled
  • Wishlist
  • Free
  • Course Includes
  • PDF slides, 19 pages
  • LaTeX source and figures (zip)
  • Quiz, 5 questions
Tags:
Self-Attention Multi-Head Attention Positional Encoding NLP Transformer



What you will learn

  • Compute scaled dot-product attention from queries, keys and values
  • Assemble a transformer block from multi-head attention, positional encoding, a feed-forward layer and residual connectio...
  • Distinguish encoder-only models such as BERT from decoder-only models such as GPT

Courselet Content

2 components
NLP - The Transformer Revolution (pdf)
395.77 K

Requirements

  • Attention in sequence-to-sequence models, matrix multiplication, softmax

General Overview

Description

Attention(Q, K, V) equals softmax(QK^T / sqrt(dk)) V, and in self-attention the queries, keys and values all come from the same sequence. Self-attention has no notion of position, so sinusoidal positional encodings are added. Multi-head attention has 8 heads of dimension 64 for a model dimension of 512. A transformer block has self-attention and a feed-forward layer of dimension 2048, each with a residual connection. Because all positions are computed in parallel, training is faster than with RNNs. Encoder-only models such as BERT and decoder-only models such as GPT belong to the transformer family tree. The encoder-decoder transformer of 2017 is treated in detail in courselet 100147, The Transformer Architecture.

Courses that include this CL

blog
Last Updated 19th November 2026
  • 5
  • Free

Recommended for you

blog
Last Updated 3rd December 2024
  • 5
  • Free
blog
Last Updated 19th November 2026
  • 8
  • Free
blog
Last Updated 19th July 2023
  • 0
  • 0
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 17th December 2022
  • 30
blog
Last Updated 16th January 2023
  • 49
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 19th November 2026
  • 7
  • Free
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/