One-hot vectors, co-occurrence counts, skip-gram, GloVe and FastText with worked numbers
|
Next Words - Lesson 4 - Word Representations (pdf)
|
388.63 K |
In a vocabulary of 50,000 words, each word has a one-hot vector, and the cosine of any two different words is 0. Counting the words around a word, the distributional hypothesis of 1954, gives cat and dog a cosine of 0.913 against 0.816 for cat and mat. An embedding has 300 learned numbers where a count row over 100,000 words has 100,000 entries. Skip-gram vectors are learned by labelling each real neighbour 1 and K = 5 noise words 0, and after one training step the score of cat and sat is 1.31, up from 1.10. The offset from man to woman added to king is nearest to queen, GloVe is a fit to log counts, and in FastText the word where is a sum of 15 vectors. Students work through the five notebooks, with the text of Pride and Prejudice as data.