Quantization to INT8, knowledge distillation, pruning, mobile architectures and the energy cost of models
|
NLP - Efficiency and Deployment (pdf)
|
517.48 K |
Large language models are over-parameterized, and a large share of their weights can be removed by magnitude pruning with little loss of accuracy. In quantization, weights are stored with 16, 8 or 4 bits, down from 32. In knowledge distillation, a small student model is trained on the soft labels of a large teacher, for example cat 0.9, tiger 0.05 and dog 0.03, at a temperature T of 3 to 5, with distillation loss and true-label loss weighted by alpha. DistilBERT and TinyBERT are examples. Deployment also depends on hardware such as GPUs, TPUs and edge TPUs and on the energy cost of training and inference. In the mobile example, BERT-base is reduced from 440 MB to 13 MB.