Encoder states, teacher forcing, beam search, nucleus sampling, BLEU and ROUGE
|
Next Words - Lesson 7 - Sequence-to-Sequence, Decoding (pdf)
|
437.18 K |
An encoder with 512 numbers per direction has 30,720 numbers for a source of 30 tokens, and a 50-token output has 30 × 50 = 1,500 attention scores. In teacher forcing, named in 1989, the prefix is the reference, and at inference a 50-token output is on the reference throughout with probability 0.364 when a token is off it with probability 0.02. Greedy decoding gives the path 0.35 × 0.40 = 0.140 where another path has 0.25 × 0.90 = 0.225, and in beam search the B best partial sequences are kept. The nucleus at p = 0.9 of one example has 4 tokens. BLEU of The cat on the mat is 40.9 and its ROUGE-1 is 0.833. Students work through the seven notebooks on sequence-to-sequence models and decoding, with the text of Pride and Prejudice as data.