Next Words - Lesson 7 - Sequence-to-Sequence, Decoding

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Next Words - Lesson 7 - Sequence-to-Sequence, Decoding

Encoder states, teacher forcing, beam search, nucleus sampling, BLEU and ROUGE

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • PDF lecture, 44 pages
  • Jupyter notebooks, 7 (zip)
  • Quiz, 5 questions
Tags:
Sequence-to-Sequence Beam Search Nucleus Sampling BLEU teacher forcing



What you will learn

  • Count encoder numbers and attention scores for a source and an output of given lengths.
  • Compute greedy, beam-search and nucleus outputs from a small tree of probabilities.
  • Compute precision, recall, BLEU and ROUGE-1 of a candidate against a reference.

Courselet Content

2 components
Next Words - Lesson 7 - Sequence-to-Sequence, Decoding (pdf)
437.18 K

Requirements

  • Probability, softmax, attention, Python

General Overview

Description

An encoder with 512 numbers per direction has 30,720 numbers for a source of 30 tokens, and a 50-token output has 30 × 50 = 1,500 attention scores. In teacher forcing, named in 1989, the prefix is the reference, and at inference a 50-token output is on the reference throughout with probability 0.364 when a token is off it with probability 0.02. Greedy decoding gives the path 0.35 × 0.40 = 0.140 where another path has 0.25 × 0.90 = 0.225, and in beam search the B best partial sequences are kept. The nucleus at p = 0.9 of one example has 4 tokens. BLEU of The cat on the mat is 40.9 and its ROUGE-1 is 0.833. Students work through the seven notebooks on sequence-to-sequence models and decoding, with the text of Pride and Prejudice as data.

Courses that include this CL

blog
Last Updated 20th November 2026
  • 1
  • Free

Recommended for you

blog
Last Updated 3rd December 2024
  • 5
  • Free
blog
Last Updated 8th March 2025
  • 1
blog
Last Updated 22nd February 2025
  • 1
blog
Last Updated 3rd May 2024
  • 17
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 20th May 2025
  • 2
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 24th September 2026
  • 5
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/