Gen AI

Embeddings

Turning words into coordinates, so that things with similar meanings end up close together and a computer can measure how related they are.

Words plotted as points in a vector space, where man to king points in the same direction as woman to queen

Why Does This Exist?

Tokenization leaves us with integer ids like 883. But those numbers are arbitrary labels: token 883 isn't "more" than token 882, and nothing about the number tells the model that "dog" and "puppy" are related while "dog" and "algebra" aren't. Raw ids carry no meaning.

Embeddings fix this. Each token id is mapped to a vector, a list of numbers, positioned in a high-dimensional space so that meaning becomes geometry. Words used in similar contexts land near each other. Suddenly the model can measure similarity with plain arithmetic: close vectors mean related meanings. This is the representation every modern language model builds on, and it's also what powers semantic search and recommendation.

Think of It Like This

A map of meaning

On a real map, cities near each other are probably a short drive apart, and the direction you travel means something too: head the same way from two different towns and you reach two similar places. An embedding space is a map of meaning. "Coffee" and "tea" sit in the same neighborhood; "Paris" and "France" are separated by roughly the same step as "Tokyo" and "Japan". Distance and direction both carry information.

How It Actually Works

An embedding layer is really just a big lookup table with one learnable row per vocabulary token. Token id 883 grabs row 883, a vector of maybe 768 numbers. Those numbers start random and are shaped during training: whenever the model's predictions are off, backpropagation nudges the vectors so tokens appearing in similar contexts drift together.

The famous payoff is that relationships show up as consistent directions. The step from "man" to "king" turns out to be about the same as the step from "woman" to "queen", so king - man + woman lands near queen. To compare two vectors we usually use cosine similarity, the angle between them: pointing the same way means similar meaning, regardless of length.

How embeddings take shape

Step 1 of 3

Look up

Each token id indexes a row in the embedding table — its starting vector.

Show all steps

Look up: Each token id indexes a row in the embedding table — its starting vector.
Train: As the model learns, vectors for tokens used in similar ways are pulled closer together.
Compare: Cosine similarity measures the angle between two vectors to score how related they are.

Show Me the Code

1import numpy as np2
3
4def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:5    # 1.0 means identical direction; 0 means unrelated; -1 means opposite.6    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))7
8
9king = np.array([0.9, 0.7, 0.1])10queen = np.array([0.8, 0.9, 0.1])11algebra = np.array([-0.6, 0.2, 0.9])12print(round(cosine_similarity(king, queen), 2))    # -> 0.97, close13print(round(cosine_similarity(king, algebra), 2))  # -> -0.19, far apart

Real embeddings have hundreds of dimensions, but the comparison is exactly this: an angle between two vectors.

Watch Out For

Embeddings inherit the biases in their data

Because vectors are learned from human text, they absorb human stereotypes along with useful patterns. Analogies like "man is to doctor as woman is to nurse" have shown up in trained embeddings. If you use them in a real product, audit for bias rather than assuming the geometry is neutral.

A static word vector can't handle multiple meanings

Classic word embeddings give "bank" one fixed vector, blending the riverbank and the money senses into a muddle. Modern transformers solve this with self-attention, producing a fresh, context-aware vector for each word each time, so "bank" shifts depending on its neighbors.

The Quick Version

Embeddings map arbitrary token ids to vectors where meaning becomes geometry.
Similar words land near each other; relationships appear as consistent directions.
The embedding layer is a learnable lookup table, shaped by training.
Cosine similarity scores relatedness by the angle between two vectors.
Watch for inherited bias, and remember static vectors can't disambiguate meanings on their own.