NLP - Efficiency and Deployment

  • 0 Rating
  • 0 Reviews
  • 2 Students Enrolled

NLP - Efficiency and Deployment

Quantization to INT8, knowledge distillation, pruning, mobile architectures and the energy cost of models

  • 0 Rating
  • 0 Reviews
  • 2 Students Enrolled
  • Wishlist
  • Free
  • Course Includes
  • PDF slides, 21 pages
  • LaTeX source and figures (zip)
  • Quiz, 4 questions
Tags:
Efficiency Quantization Knowledge Distillation Pruning NLP



What you will learn

  • Apply post-training quantization from FP32 to INT8
  • Train a student model on the soft labels of a teacher at temperature T, with distillation and true-label loss weighted b...
  • Prune weights by magnitude and describe the effect on model size and speed

Courselet Content

2 components
NLP - Efficiency and Deployment (pdf)
517.48 K

Requirements

  • Transformer models, neural network training with PyTorch

General Overview

Description

Large language models are over-parameterized, and a large share of their weights can be removed by magnitude pruning with little loss of accuracy. In quantization, weights are stored with 16, 8 or 4 bits, down from 32. In knowledge distillation, a small student model is trained on the soft labels of a large teacher, for example cat 0.9, tiger 0.05 and dog 0.03, at a temperature T of 3 to 5, with distillation loss and true-label loss weighted by alpha. DistilBERT and TinyBERT are examples. Deployment also depends on hardware such as GPUs, TPUs and edge TPUs and on the energy cost of training and inference. In the mobile example, BERT-base is reduced from 440 MB to 13 MB.

Courses that include this CL

blog
Last Updated 19th November 2026
  • 5
  • Free

Recommended for you

blog
Last Updated 3rd May 2024
  • 17
blog
Last Updated 10th December 2023
  • 5
blog
Last Updated 14th November 2023
  • 0
  • 0
blog
Last Updated 19th November 2026
  • 8
  • Free
blog
Last Updated 30th July 2023
  • 2
blog
Last Updated 19th July 2023
  • 0
  • 0
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 16th January 2023
  • 49
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 19th November 2023
  • 7
blog
Last Updated 19th November 2026
  • 7
  • Free
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/