Greedy decoding, beam search, temperature, top-k and top-p sampling and repetition penalties
|
NLP - Decoding Strategies (pdf)
|
466.13 K |
A language model gives a probability for each possible next word, and the text produced depends on the decoding method. In greedy decoding the most probable word is always taken, and the same input gives the same output. In beam search the k best paths are kept and scored by the sum of log probabilities, with a beam of 4 to 5 common in translation. Controlled randomness is added by sampling with temperature, top-k and top-p, where the next word is drawn from the smallest set of words with cumulative probability p. Repetition penalties of about 1.2 are applied to recent tokens. Suitable settings depend on the task, for example T = 0.2 with p = 0.95 for code. The same methods are treated in courselets 1019 (a lab) and 1021 (a deck).