Greedy, beam, temperature, top-k, nucleus and contrastive search, matched to six problems of generated text
|
NLP Decoding Strategies (pdf)
|
2.45 M |
The probability distribution of a language model has about 50,000 entries at every step, and the quality of generated text depends on how a word is chosen from it. The methods compared are greedy decoding, beam search, temperature sampling with exp(zi / T), top-k sampling, nucleus (top-p) sampling and contrastive search. They are matched to six problems of generated text, among them repetition loops, missing diversity and junk in the tail of the distribution. In contrastive search the candidate with the highest score is chosen, where score equals probability minus alpha times the largest cosine similarity to earlier tokens. Temperature is worked through on the logits 2.0, 1.0, 0.5 and 0.2, and the selection of a method is supported by a decision tree and a table of settings per task.