Generative AI
Embeddings & Vector Space
See how geometric distance and direction encode meaning in high-dimensional space.
Words as points in high-dimensional space where distance means semantic similarity and directions carry conceptual meaning.
Stage 1 of 4: Space and Similarity
Viewing pca projection. Pitch 0.20, Yaw 0.79.
- Word vector
Words are embedded as 3D points. Drag to orbit the projection. Points close together share similar meanings. The axes have no intrinsic labels; meaning is entirely relative.
Challenge
Find a word combination in the vocabulary that results in a target concept.
Navigate the arithmetic bar to find a path from one concept group to another.
Challenge not yet solved.
Check your understanding
2 questions in the bank. Each attempt draws a fresh set in a fresh order, so a second go is a real second go.
Neural networks don't process letters or strings. To a language model, words are points in a high-dimensional space — an embedding.
When a model learns to embed words, it learns to place similar concepts near each other. More importantly, the directions between those points encode semantic meaning, which is why we can do math with words. If the vector from "man" to "woman" encodes the concept of gender, applying that exact same translation to "king" lands perfectly in the region of "queen".
This lab simulates a small, 3-dimensional embedding space so you can explore how geometric distance becomes conceptual meaning.
Reference
- Vector
- An array of numbers representing a point in space.
- Cosine Similarity
- cos(θ) = A·B / (||A|| ||B||) — Measures angle between vectors. 1 is identical, 0 is orthogonal, -1 is opposite.
- Dot Product
- A·B = Σ(a_i * b_i) — Projection of one vector onto another.
- Semantic Arithmetic
- v(king) - v(man) + v(woman) ≈ v(queen)
Break it on purpose
Try viewing the space from an edge-on angle (Pitch 0, Yaw 0) in PCA projection. Distinct clusters will suddenly appear to overlap and merge. This "breaks" the visualization because reducing high-dimensional meaning down to a 2D screen causes a loss of information — points that look close together here might actually be far apart in the true embedding space.