Next Words - Lesson 11 - Tokenization, Data at Scale

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Next Words - Lesson 11 - Tokenization, Data at Scale

Word-level, BPE, WordPiece and Unigram vocabularies, token premiums, web corpora

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • PDF lecture, 36 pages
  • Jupyter notebooks, 5 (zip)
  • Quiz, 5 questions
Tags:
Tokenization Vocabulary BPE web corpora deduplication



What you will learn

  • Estimate the share of new next words in a text from the number of word types seen once.
  • Count the merges, the entries and the added embedding parameters of a BPE vocabulary.
  • Compute the premium of a language as tokens over English tokens and the share of English text in one context window.

Courselet Content

2 components
Next Words - Lesson 11 - Tokenization, Data at Scale (pdf)
391.13 K

Requirements

  • Probability, embeddings, Python

General Overview

Description

GPT-2 gives Bulgarian 5.51 times the English tokens and Chinese 3.21 times. In a word-level vocabulary, every unlisted word is mapped to one unknown symbol, and 2,033 of the 6,274 word types in a corpus of 114,328 tokens occur once. In byte-pair encoding, the most frequent adjacent pair is merged and one token is added per merge, shown on a five-word corpus whose first merge has count 11. WordPiece, Unigram and SentencePiece are compared with BPE, and byte-level BPE has a base of 256 bytes. A vocabulary of 128,000 entries has 393 million more parameters than one of 32,000 at d = 4,096. Of Common Crawl, 11.67 per cent is kept in RefinedWeb, and after deduplication of C4 memorised output is 0.189 per cent against 1.926 per cent. Students work through five notebooks on tokenization and data at scale, with the Pride and Prejudice text as data.

Courses that include this CL

blog
Last Updated 20th November 2026
  • 1
  • Free

Recommended for you

blog
Last Updated 22nd February 2025
  • 1
blog
Last Updated 3rd May 2024
  • 17
blog
Last Updated 10th December 2023
  • 5
blog
Last Updated 14th November 2023
  • 0
  • 0
blog
Last Updated 30th July 2023
  • 2
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 17th December 2022
  • 4
blog
Last Updated 16th January 2023
  • 49
  • Free
blog
Last Updated 20th May 2025
  • 2
blog
Last Updated 24th September 2026
  • 5
blog
Last Updated 19th November 2023
  • 7
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/