NLP - The Transformer Revolution, Notebook

  • 0 Rating
  • 0 Reviews
  • 0 Students Enrolled

NLP - The Transformer Revolution, Notebook

A small transformer built in PyTorch on 200 Python snippets, with the mathematics of attention

  • 0 Rating
  • 0 Reviews
  • 0 Students Enrolled
  • Wishlist
  • Free
  • Course Includes
  • Week 5 notebook as PDF and .ipynb
  • Quiz, 4 questions
Tags:
PyTorch Code Completion NLP Transformer Self-Attention



What you will learn

  • Implement scaled dot-product attention, multi-head attention and sinusoidal positional encoding in PyTorch
  • Assemble a transformer block and train a small code-completion model on 200 Python code snippets
  • Compare the time and memory cost of self-attention with that of an RNN

Requirements

  • Python with PyTorch, attention in sequence-to-sequence models, matrix multiplication

General Overview

Description

Code completion is next-word prediction over program text, and a small transformer can be trained for it on 200 Python code snippets. Students implement scaled dot-product attention, multi-head attention, sinusoidal positional encoding, a feed-forward layer, a transformer block and a code transformer in PyTorch. Attention(Q, K, V) equals softmax(QK^T / sqrt(dk)) V, and the effect of the scaling by sqrt(dk) is demonstrated numerically. Self-attention has time cost O(n^2 d) and memory cost O(n^2), against O(n d^2) and O(d) for an RNN. Masked self-attention for generation, relative position encoding and sparse attention are the advanced techniques, and a function-name predictor with 2 layers and 4 heads is the closing exercise.

Recommended for you

blog
Last Updated 3rd December 2024
  • 5
  • Free
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 17th December 2022
  • 30
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 20th May 2025
  • 2
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 3rd October 2025
  • 97
  • Free
blog
Last Updated 7th November 2022
  • 157
  • Free
blog
Last Updated 7th November 2022
  • 23
  • Free
blog
Last Updated 15th January 2026
  • 102
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free
blog
Last Updated 20th May 2025
  • 31
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/