Skip to content
AI360Xpert
Beta
Core ML

Visual explainer

Word2Vec

A sliding window turns co-occurrence into vectors, so scores 2, 1, 0 and minus 1 become probabilities and neighbors land near each other.

One-hot keeps apple as far from orange as from car.
One-hot keeps apple as far from orange as from car.

One-hot gives every word its own axis, so apple sits 1.41 from orange and 1.41 from car. Every pair is equally far, which means the vectors know nothing about meaning yet at all.

Window Slides On

A window of two frames fox with its four neighbors.
A window of two frames fox with its four neighbors.

You slide a window of two across the sentence and read fox with quick, brown, jumps and lazy beside it. Words that share windows will soon share neighborhoods everywhere.

Context Becomes Odds

Scores 2, 1, 0 and minus 1 become odds 0.644 down to 0.032.
Scores 2, 1, 0 and minus 1 become odds 0.644 down to 0.032.

Fox scores jumps at 2, brown at 1, lazy at 0 and moon at minus 1. Softmax turns those into 0.644, 0.237, 0.087 and 0.032. The 0.644 target pulls its vector closer; the 0.032 stranger barely moves.

Average or Predict

CBOW averages four context vectors into 2, 2 while skip-gram predicts outward.
CBOW averages four context vectors into 2, 2 while skip-gram predicts outward.

CBOW averages four context vectors into [2, 2] and guesses the middle fast. Skip-gram flips it, spending more compute to predict outward from one word.

Neighbors Appear

Apple reaches orange at 0.86 but car at only 0.12.
Apple reaches orange at 0.86 but car at only 0.12.

After training, apple reaches orange at 0.86 but car at only 0.12. Same axes as one-hot, yet distance finally means something you can use for search and ranking.

Speed or Rarity

Big corpus defaults to CBOW while rare words pay for skip-gram.
Big corpus defaults to CBOW while rare words pay for skip-gram.

CBOW trains fast on frequent words, so default to it for big corpora. When rare words matter and data is small, pay for skip-gram instead.

Where It Breaks

One bank vector sits stranded between river and money.
One bank vector sits stranded between river and money.

Bank gets one vector, so river-bank and money-bank pull it to the middle where neither lives. One address cannot house two meanings, so contextual models later gave each word a neighborhood instead.

The Quick Version

  • One-hot keeps every pair equally far.
  • A window of two frames each word.
  • Scores become 0.644, 0.237, 0.087, 0.032.
  • CBOW averages; skip-gram predicts outward.
  • Apple nears orange at 0.86.
  • Big corpus: CBOW; rare words: skip-gram.
  • One vector cannot hold two meanings.