Encoder-decoder models, the context-vector bottleneck, attention and BLEU for translation
|
NLP - Sequence-to-Sequence Models (pdf)
|
431.84 K |
Input and output have different lengths, as with the French Je t'aime and the German Ich liebe dich, two words against three. In an encoder-decoder model the whole input is encoded into a fixed-size context vector, and the output is generated one word at a time from that vector, between a START and an END token. With teacher forcing, true words are fed to the decoder during training. A long sentence is compressed into one vector with loss, and translation quality is lower beyond about 30 tokens. In attention, all encoder hidden states are kept and weighted at every decoding step, which gives interpretable alignments between source and target words. Translations are scored with BLEU, and beam search is used in decoding.