Next Words - Lesson 11 - Tokenization, Data at Scale

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Next Words - Lesson 11 - Tokenization, Data at Scale

Word-level, BPE, WordPiece and Unigram vocabularies, token premiums, web corpora

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • PDF lecture, 36 pages
  • Jupyter notebooks, 5 (zip)
  • Quiz, 5 questions
Tags:
Tokenization Vocabulary BPE web corpora deduplication



What you will learn

  • Estimate the share of new next words in a text from the number of word types seen once.
  • Count the merges, the entries and the added embedding parameters of a BPE vocabulary.
  • Compute the premium of a language as tokens over English tokens and the share of English text in one context window.

Courselet Content

2 components
Next Words - Lesson 11 - Tokenization, Data at Scale (pdf)
391.13 K

Requirements

  • Probability, embeddings, Python

General Overview

Description

GPT-2 gives Bulgarian 5.51 times the English tokens and Chinese 3.21 times. In a word-level vocabulary, every unlisted word is mapped to one unknown symbol, and 2,033 of the 6,274 word types in a corpus of 114,328 tokens occur once. In byte-pair encoding, the most frequent adjacent pair is merged and one token is added per merge, shown on a five-word corpus whose first merge has count 11. WordPiece, Unigram and SentencePiece are compared with BPE, and byte-level BPE has a base of 256 bytes. A vocabulary of 128,000 entries has 393 million more parameters than one of 32,000 at d = 4,096. Of Common Crawl, 11.67 per cent is kept in RefinedWeb, and after deduplication of C4 memorised output is 0.189 per cent against 1.926 per cent. Students work through five notebooks on tokenization and data at scale, with the Pride and Prejudice text as data.

Courses that include this CL

blog
Last Updated 20th November 2026
  • 1
  • Free

Recommended for you

blog
Last Updated 8th March 2025
  • 1
blog
Last Updated 15th January 2025
  • 9
blog
Last Updated 10th December 2023
  • 5
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 27th February 2026
  • 13
  • Free
blog
Last Updated 5th June 2025
  • 3
  • Free
blog
Last Updated 16th January 2023
  • 49
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 31st July 2025
  • 32
blog
Last Updated 24th September 2026
  • 5
blog
Last Updated 20th August 2025
  • 50
  • Free
blog
Last Updated 19th November 2023
  • 7
blog
Last Updated 13th December 2022
  • 141
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/