NLP - Pre-trained Language Models

  • 0 Rating
  • 0 Reviews
  • 2 Students Enrolled

NLP - Pre-trained Language Models

Masked language modeling in BERT, autoregressive pre-training in GPT and fine-tuning for a task

  • 0 Rating
  • 0 Reviews
  • 2 Students Enrolled
  • Wishlist
  • Free
  • Course Includes
  • PDF slides, 20 pages
  • LaTeX source and figures (zip)
  • Quiz, 4 questions
Tags:
Pre-training BERT GPT Fine-tuning NLP



What you will learn

  • Explain masked language modeling with the 15 per cent masking rule and its 80, 10 and 10 per cent split
  • Compare the bidirectional encoder of BERT with the autoregressive decoder of GPT
  • Fine-tune a pre-trained model for a classification task by adding a task-specific head

Courselet Content

2 components
NLP - Pre-trained Language Models (pdf)
361.71 K

Requirements

  • Transformer architecture, softmax and cross-entropy, basic PyTorch

General Overview

Description

Pre-training on raw text is self-supervised, since the labels are taken from the text itself. BERT is a transformer encoder trained with masked language modeling. Of all tokens, 15 per cent are selected, and of these 80 per cent are replaced by [MASK], 10 per cent by a random token and 10 per cent are left unchanged. GPT is a transformer decoder trained autoregressively on the next word. BERT-base has 12 layers, 768 hidden units and 110 million parameters, BERT-large 24 layers, 1024 hidden units and 340 million. A pre-trained model is fine-tuned with a task-specific head on a small labelled data set and evaluated on benchmarks such as GLUE, with nine language understanding tasks.

Courses that include this CL

blog
Last Updated 19th November 2026
  • 5
  • Free

Recommended for you

blog
Last Updated 10th December 2023
  • 5
blog
Last Updated 30th July 2023
  • 2
blog
Last Updated 25th July 2023
  • 2
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 17th December 2022
  • 30
blog
Last Updated 16th January 2023
  • 49
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 7th November 2022
  • 88
  • Free
blog
Last Updated 19th November 2023
  • 7
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/