Core ML

Sentence Embeddings

An embedding technique that maps entire sentences or paragraphs into a single dense vector, allowing algorithms to compare the semantic meaning of large blocks of text.

Why Does This Exist?

Word2Vec gave us a way to mathematically compare words. But in the real world, we rarely compare isolated words. We need to compare queries to documents, questions to answers, or tweets to other tweets.

If you have vectors for individual words, how do you get a vector for a 10-word sentence?

The naive approach is averaging. If you take the vectors for "The", "dog", "bit", "the", "man" and average them together, you get a single vector. But this approach is disastrous for semantics. The average vector for "The man bit the dog" is exactly the same as "The dog bit the man". All syntax, word order, and nuanced meaning is obliterated.

Sentence embeddings solve this by training a model to directly output a single vector representing the meaning of an entire sentence, preserving the syntax and order.

Think of It Like This

Imagine trying to describe a beautiful landscape painting (the sentence) by taking the average of all the colors (the words) used on the canvas.

If you average a beautiful sunset painting, you might just get a muddy brown color. If you average a painting of a desert, you might get the exact same muddy brown. You have lost all the structure.

Sentence embeddings don't average the paint colors. They look at the painting as a cohesive whole and assign it a coordinate in an art gallery, placing it right next to other landscape paintings, regardless of the specific paint colors used.

How It Actually Works

While early approaches like Doc2Vec extended Word2Vec, the modern standard for sentence embeddings is the Siamese Network architecture, most famously implemented by Sentence-BERT (SBERT).

The Siamese Architecture

A Siamese network uses two identical copies of the same underlying model (like BERT) sharing the exact same weights.

Pass Sentence A into the first model. It outputs a dense vector (e.g., length 384).
Pass Sentence B into the second model. It outputs a dense vector.
Compute the Cosine Similarity between the two vectors.

Contrastive Training

To teach the model how to embed sentences properly, we train it on pairs of sentences that are either related or unrelated (e.g., pairs from the SNLI dataset, which labels sentence pairs as contradiction, entailment, or neutral).

If the sentences mean the same thing, the loss function forces their output vectors closer together in the high-dimensional space. If they contradict, it pushes them apart.

By forcing the model to optimize for cosine similarity across entire sentences, the final vector becomes a highly compressed representation of the sentence's holistic meaning, rather than a muddy average of its parts.

Show Me the Code

In Python, the sentence-transformers library (built on top of Hugging Face) is the standard for generating sentence embeddings.

1from sentence_transformers import SentenceTransformer, util2
3# Load a pre-trained sentence embedding model4model = SentenceTransformer('all-MiniLM-L6-v2')5
6# Define our sentences7sentences = [8    "A man is playing a guitar.",9    "A guy is strumming an acoustic instrument.",10    "A woman is eating a sandwich."11]12
13# Generate the embeddings (outputs a matrix of shape 3 x 384)14embeddings = model.encode(sentences)15
16# Compute cosine similarity between sentence 0 and 117sim_0_1 = util.cos_sim(embeddings[0], embeddings[1])18print(f"Similarity (Guitar vs Strumming): {sim_0_1.item():.4f}")19# -> 0.7765 (High similarity, even though they share almost no exact words!)20
21# Compute cosine similarity between sentence 0 and 222sim_0_2 = util.cos_sim(embeddings[0], embeddings[2])23print(f"Similarity (Guitar vs Sandwich): {sim_0_2.item():.4f}")24# -> -0.0125 (No similarity)

Watch Out For

Asymmetric Search (Queries vs Documents). Standard sentence embeddings are trained symmetrically (comparing a sentence to another sentence). If you use them for Semantic Search, where the input is a short query (e.g., "python syntax") and the target is a long document, symmetric models perform poorly. You need models explicitly trained for asymmetric search (like msmarco models), which learn to map short questions to long answers.

The Quick Version

Averaging individual word vectors destroys syntax and order, resulting in poor representations for sentences.
Sentence embeddings map an entire sequence of text into a single, unified vector.
Modern sentence embeddings (like SBERT) use a Siamese network architecture, encoding two sentences independently and optimizing their cosine similarity.
They allow for lightning-fast semantic comparison, making them the foundational technology behind modern Vector Databases and RAG systems.