NLP - Neural Language Models (pdf)
Word embeddings, skip-gram Word2Vec with negative sampling and cosine similarity of words
|
NLP - Neural Language Models (pdf)
|
301.68 K |
Words that occur in similar contexts have similar meanings, and the vectors of such words lie close together. A one-hot ID over a 50,000-word vocabulary has 50,000 entries, a dense embedding 100 to 300. In skip-gram Word2Vec, context words are predicted from a center word, with two embedding matrices and negative sampling. Similarity is measured by cosine distance, and vector arithmetic such as king - man + woman gives queen. Embeddings are evaluated on semantic and syntactic analogy questions. A single vector per word is an average over the senses of bank, has no word order and is tied to a fixed vocabulary.