NLP - Tokenization and Subword Models

  • 0 Rating
  • 0 Reviews
  • 2 Students Enrolled

NLP - Tokenization and Subword Models

Byte pair encoding, WordPiece and SentencePiece, vocabulary size and tokenization across languages

  • 0 Rating
  • 0 Reviews
  • 2 Students Enrolled
  • Wishlist
  • Free
  • Course Includes
  • PDF slides, 18 pages
  • LaTeX source and figures (zip)
  • Quiz, 4 questions
Tags:
Tokenization Byte Pair Encoding Subword Models Vocabulary NLP



What you will learn

  • Build a byte pair encoding vocabulary by merging the most frequent adjacent pair again and again
  • Compare BPE and WordPiece tokenizations of the same text
  • Train a tokenizer on a domain corpus with different vocabulary sizes and compare the sequence lengths

Courselet Content

2 components
NLP - Tokenization and Subword Models (pdf)
335.17 K

Requirements

  • Language modeling basics, Python strings and dictionaries

General Overview

Description

A character-level vocabulary has about 100 symbols and very long sequences, a word-level vocabulary more than 170,000 entries and no answer for unseen words. In byte pair encoding, the initial vocabulary is the set of characters and the most frequent adjacent pair is merged repeatedly, typically 10,000 to 50,000 times. Among the learned merges are suffixes such as er and est. In WordPiece, used by BERT, continuation pieces are marked with ##, and byte-level BPE is used from GPT-2 onwards. A larger vocabulary gives shorter sequences at the price of more memory, and token counts for the same text depend on the language. Accuracy depends on using the tokenizer a model was trained with.

Courses that include this CL

blog
Last Updated 19th November 2026
  • 5
  • Free

Recommended for you

blog
Last Updated 3rd December 2024
  • 5
  • Free
blog
Last Updated 3rd May 2024
  • 17
blog
Last Updated 19th November 2026
  • 8
  • Free
blog
Last Updated 19th July 2023
  • 0
  • 0
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 17th December 2022
  • 30
blog
Last Updated 16th January 2023
  • 49
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 19th November 2026
  • 7
  • Free
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/