NLP - Decoding Strategies, From Probabilities to Text

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

NLP - Decoding Strategies, From Probabilities to Text

Greedy, beam, temperature, top-k, nucleus and contrastive search, matched to six problems of generated text

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Wishlist
  • Free
  • Course Includes
  • PDF slides, 67 pages
  • LaTeX source (zip)
  • Quiz, 4 questions
Tags:
Nucleus Sampling Contrastive Search NLP Decoding Beam Search



What you will learn

  • Compute temperature-scaled softmax probabilities for a vector of logits
  • Apply top-k and nucleus filtering to a probability distribution and explain why the size of the nucleus depends on the s...
  • Select a decoding method and its parameters for translation, factual answers, code, dialogue and long stories

Courselet Content

2 components
NLP Decoding Strategies (pdf)
2.45 M

Requirements

  • Softmax, conditional probability, transformer language models

General Overview

Description

The probability distribution of a language model has about 50,000 entries at every step, and the quality of generated text depends on how a word is chosen from it. The methods compared are greedy decoding, beam search, temperature sampling with exp(zi / T), top-k sampling, nucleus (top-p) sampling and contrastive search. They are matched to six problems of generated text, among them repetition loops, missing diversity and junk in the tail of the distribution. In contrastive search the candidate with the highest score is chosen, where score equals probability minus alpha times the largest cosine similarity to earlier tokens. Temperature is worked through on the logits 2.0, 1.0, 0.5 and 0.2, and the selection of a method is supported by a decision tree and a table of settings per task.

Courses that include this CL

blog
Last Updated 19th November 2026
  • 5
  • Free

Recommended for you

blog
Last Updated 19th November 2026
  • 8
  • Free
blog
Last Updated 19th July 2023
  • 0
  • 0
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 17th December 2022
  • 30
blog
Last Updated 16th January 2023
  • 47
  • Free
blog
Last Updated 16th January 2023
  • 49
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 19th November 2026
  • 7
  • Free
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/