Next Words - Lesson 4 - Word Representations

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled

Next Words - Lesson 4 - Word Representations

One-hot vectors, co-occurrence counts, skip-gram, GloVe and FastText with worked numbers

  • 0 Rating
  • 0 Reviews
  • 1 Students Enrolled
  • Free
  • Course Includes
  • PDF lecture, 25 pages
  • Jupyter notebooks, 5 (zip)
  • Quiz, 5 questions
Tags:
Word Embeddings Word2Vec skip-gram cosine similarity negative sampling



What you will learn

  • Compute the cosine of two rows of co-occurrence counts and rank words by it.
  • Compute one skip-gram training step with negative sampling from a real pair and a noise pair.
  • Count the skip-gram pairs of a sentence and the vectors that are summed for a word in FastText.

Courselet Content

2 components
Next Words - Lesson 4 - Word Representations (pdf)
388.63 K

Requirements

  • Vectors, dot product, probability, Python

General Overview

Description

In a vocabulary of 50,000 words, each word has a one-hot vector, and the cosine of any two different words is 0. Counting the words around a word, the distributional hypothesis of 1954, gives cat and dog a cosine of 0.913 against 0.816 for cat and mat. An embedding has 300 learned numbers where a count row over 100,000 words has 100,000 entries. Skip-gram vectors are learned by labelling each real neighbour 1 and K = 5 noise words 0, and after one training step the score of cat and sat is 1.31, up from 1.10. The offset from man to woman added to king is nearest to queen, GloVe is a fit to log counts, and in FastText the word where is a sum of 15 vectors. Students work through the five notebooks, with the text of Pride and Prejudice as data.

Courses that include this CL

blog
Last Updated 20th November 2026
  • 1
  • Free

Recommended for you

blog
Last Updated 3rd December 2024
  • 5
  • Free
blog
Last Updated 3rd May 2024
  • 17
blog
Last Updated 19th November 2026
  • 8
  • Free
blog
Last Updated 16th June 2023
  • 7
blog
Last Updated 8th May 2023
  • 27
blog
Last Updated 16th January 2023
  • 49
  • Free
blog
Last Updated 7th January 2023
  • 8
  • Free
blog
Last Updated 20th May 2025
  • 2
blog
Last Updated 14th March 2025
  • 52
  • Free
blog
Last Updated 19th November 2026
  • 7
  • Free
blog
Last Updated 20th August 2025
  • 50
  • Free
blog
Last Updated 7th November 2022
  • 16
  • Free
blog
Last Updated 21st March 2025
  • 251
  • Free

Meet the instructors !

instructor
About the Instructor

https://www.joergosterrieder.com/