Masked language modeling in BERT, autoregressive pre-training in GPT and fine-tuning for a task
|
NLP - Pre-trained Language Models (pdf)
|
361.71 K |
Pre-training on raw text is self-supervised, since the labels are taken from the text itself. BERT is a transformer encoder trained with masked language modeling. Of all tokens, 15 per cent are selected, and of these 80 per cent are replaced by [MASK], 10 per cent by a random token and 10 per cent are left unchanged. GPT is a transformer decoder trained autoregressively on the next word. BERT-base has 12 layers, 768 hidden units and 110 million parameters, BERT-large 24 layers, 1024 hidden units and 340 million. A pre-trained model is fine-tuned with a task-specific head on a small labelled data set and evaluated on benchmarks such as GLUE, with nine language understanding tasks.