Byte pair encoding, WordPiece and SentencePiece, vocabulary size and tokenization across languages
|
NLP - Tokenization and Subword Models (pdf)
|
335.17 K |
A character-level vocabulary has about 100 symbols and very long sequences, a word-level vocabulary more than 170,000 entries and no answer for unseen words. In byte pair encoding, the initial vocabulary is the set of characters and the most frequent adjacent pair is merged repeatedly, typically 10,000 to 50,000 times. Among the learned merges are suffixes such as er and est. In WordPiece, used by BERT, continuation pieces are marked with ##, and byte-level BPE is used from GPT-2 onwards. A larger vocabulary gives shorter sequences at the price of more memory, and token counts for the same text depend on the language. Accuracy depends on using the tokenizer a model was trained with.