Embeddings & Vector Space

Neural networks do not process letters. To a language model a word is a point in a space of a few hundred to a few thousand dimensions — an embedding — and everything the model knows about that word is its position relative to every other word.

Two consequences, and this lab is about both. Nearby points mean similar things, so similarity becomes a measurement rather than a judgement. And the directions between points carry meaning of their own, which is why you can do arithmetic with words.

The space here is three-dimensional and hand-built, twenty words in five groups, so that the whole thing fits on a screen. Everything computed from it is real: cosine similarities, the PCA basis, the analogy.

Similarity is an angle

Cosine similarity is the cosine of the angle between two vectors: A·B / (‖A‖‖B‖). It is 1 when they point the same way, 0 when they are perpendicular, −1 when they are opposed. It ignores length entirely, which is usually what you want — a word being "more strongly" itself should not make it less similar to its neighbours.

Pin a word and the lab reports its three nearest, with the number that ranked them. dog gives cat at 0.980, puppy at 0.968, kitten at 0.957.

Directions have meanings

If the step from man to woman is a direction that means "change the gender", then adding that same step to king should land somewhere near queen. It does: king − man + woman produces a point that is no word at all, and the nearest actual word to it is queen, at similarity 0.996.

The lab computes that rather than assuming it. The operands are excluded from the search, which is not a technicality — without it an analogy will happily return one of its own inputs and you will have learned nothing.

paris − france + italy is the same trick on a different relation, and it lands on rome exactly.

What the picture costs

You are looking at three dimensions through a two-dimensional window, and one direction has to go. Whichever direction you are looking along is the one you cannot see — so the more the cloud varies along your line of sight, the more the picture is hiding.

Variance on screen is that, as a number: the fraction of the cloud's total variance carried by the two axes you can see. And the compressed pair names the current casualty — the two words whose real distance this view understates by the largest factor.

Note that there is always one. At the clearest available angle it is woman and queen, understated forty-sevenfold. No flat picture of a curved-up space is faithful to every pair at once; the readout tells you which pair you are currently being misled about, not that a perfect view exists.

PCA picks the best available compromise: the two orthogonal directions along which the data varies most, so the picture discards the least total variance. At yaw 0 that is 87% on screen. Turn to look straight down the first principal component — yaw ±90° — and it collapses to 34%, whole clusters slide behind each other, and kitten and italy land a fifth of a unit apart while being four and a half apart in the embedding.

The one thing the projection may not do

Watch the readouts while you do all that. They do not move.

Nearest neighbours and similarities in this lab are computed from the embedding, never from the picture — and that is a correctness requirement, not a preference. Cosine similarity survives rotating a space and does not survive shifting it, because it is an angle measured at the origin. A PCA projection subtracts the mean, so it moves the origin. Compute neighbours in that basis instead and you get answers like girl's nearest neighbour being france, which is not a fact about language.

This is the habit worth taking away. Every t-SNE or UMAP plot of an embedding you have ever seen is one of these views. The clusters in it are real evidence and the distances in it are not measurements — if you want a distance, ask the vectors.