Skip to content
AI360Xpert
Core ML

Word2Vec

A neural network technique that converts words into dense vectors, placing words with similar meanings close together in a mathematical space.

Word2Vec forces words that appear in similar contexts to have similar vector representations, unlocking semantic arithmetic like King - Man + Woman = Queen.
Word2Vec forces words that appear in similar contexts to have similar vector representations, unlocking semantic arithmetic like King - Man + Woman = Queen.

Why Does This Exist?

Previous text representations like Bag of Words and TF-IDF represent words as independent, orthogonal dimensions. In those systems, the vector for "cat" is completely unrelated to the vector for "dog". The mathematical distance between "cat" and "dog" is exactly the same as the distance between "cat" and "skyscraper".

This is fundamentally flawed. In reality, "cat" and "dog" share a massive amount of semantic overlap: they are both pets, they both have fur, and they both chase things.

Word2Vec was a massive breakthrough in 2013 because it successfully captured this semantic meaning. It compresses words into dense vectors (usually 100 to 300 dimensions) where geometrically close vectors represent semantically similar words.

Think of It Like This

Think of It Like This

Imagine trying to map every person in your city onto a 2D coordinate plane, but the only rule is: "People who hang out at the same places must be placed close together."

You look at a coffee shop and see Alice and Bob. You move their pins closer. You look at a gym and see Bob and Charlie. You move their pins closer. After scanning every location in the city, the map organizes itself. All the fitness enthusiasts end up clustered in one quadrant, and all the coffee lovers in another.

Word2Vec does this with words. Its guiding philosophy is: "You shall know a word by the company it keeps." If "cat" and "dog" both frequently appear next to words like "pet", "fed", and "barked", Word2Vec will force their coordinates to be close together in the vector space.

How It Actually Works

Word2Vec is not a deep neural network; it is a shallow, two-layer neural network trained on a fake task. Once the training is done, we throw away the network and keep the learned weights (the embeddings).

There are two primary architectures to train Word2Vec:

1. Continuous Bag of Words (CBOW)

The model looks at the surrounding context words and tries to predict the missing target word in the middle.

  • Input context: ["the", "cat", "on", "the"]
  • Target to predict: "sat"

CBOW is faster to train and works well for frequent words.

2. Skip-Gram

The exact opposite of CBOW. The model takes a single target word and tries to predict the surrounding context words.

  • Input target: "sat"
  • Context to predict: ["the", "cat", "on", "the"]

Skip-gram is slower but performs much better on rare words, because each rare word gets its own dedicated prediction step.

The Magic of Vector Arithmetic

Because Word2Vec forces semantic relationships into a geometric space, directions in that space represent actual linguistic concepts.

The most famous example is gender and royalty. If you take the vector for King, subtract the vector for Man, and add the vector for Woman, the resulting coordinate lands almost exactly on the vector for Queen. King⃗−Man⃗+Woman⃗≈Queen⃗\vec{King} - \vec{Man} + \vec{Woman} \approx \vec{Queen}

The model learned the concept of "gender" and "royalty" completely unsupervised, just by reading millions of sentences.

Show Me the Code

In Python, the gensim library is the standard tool for training and querying Word2Vec models.

from gensim.models import Word2Vec
# A tiny toy corpus (normally this would be millions of sentences)sentences = [    ["the", "king", "rules", "the", "kingdom"],    ["the", "queen", "rules", "the", "kingdom"],    ["the", "man", "walks", "the", "dog"],    ["the", "woman", "walks", "the", "dog"]]
# Train the model (vector_size=100 dimensions, window=2 context words)model = Word2Vec(sentences, vector_size=100, window=2, min_count=1, sg=1) # sg=1 means Skip-gram
# Get the dense vector for a wordprint("Vector for 'king':\n", model.wv['king'][:5], "...") # -> [-0.004  0.001 -0.003  0.009 -0.001] ...
# Find the most similar words mathematicallyprint("Most similar to 'king':", model.wv.most_similar('king', topn=1))# -> [('queen', 0.142)] 
# (Note: Results on this 4-sentence corpus are random. # Real semantic relationships require massive datasets.)

Watch Out For

Watch Out For

Static Embeddings. Word2Vec produces static embeddings. The word "bank" gets exactly one vector, regardless of whether the sentence is "I sat by the river bank" or "I deposited money in the bank". Modern transformer models solved this by generating dynamic, contextual embeddings that change based on the surrounding sentence.

Watch Out For

Out of Vocabulary (OOV) Words. If Word2Vec encounters a word it didn't see during training, it crashes. It cannot generate a vector for a novel word.

The Quick Version

  • Word2Vec represents words as dense, low-dimensional vectors (e.g., 300 numbers) where semantic similarity equals geometric proximity.
  • It trains a shallow neural network on a fake prediction task: either predicting a word from its context (CBOW) or predicting the context from a word (Skip-gram).
  • The resulting vectors exhibit linear arithmetic properties, like King - Man + Woman = Queen.
  • It is a static embedding model, meaning words with multiple definitions (like "bank") are forced into a single, averaged vector.

Related concepts