Word-level, BPE, WordPiece and Unigram vocabularies, token premiums, web corpora
|
Next Words - Lesson 11 - Tokenization, Data at Scale (pdf)
|
391.13 K |
GPT-2 gives Bulgarian 5.51 times the English tokens and Chinese 3.21 times. In a word-level vocabulary, every unlisted word is mapped to one unknown symbol, and 2,033 of the 6,274 word types in a corpus of 114,328 tokens occur once. In byte-pair encoding, the most frequent adjacent pair is merged and one token is added per merge, shown on a five-word corpus whose first merge has count 11. WordPiece, Unigram and SentencePiece are compared with BPE, and byte-level BPE has a base of 256 bytes. A vocabulary of 128,000 entries has 393 million more parameters than one of 32,000 at d = 4,096. Of Common Crawl, 11.67 per cent is kept in RefinedWeb, and after deduplication of C4 memorised output is 0.189 per cent against 1.926 per cent. Students work through five notebooks on tokenization and data at scale, with the Pride and Prejudice text as data.