Gen AI

Semantic Chunking

Instead of splitting documents at arbitrary character limits, semantic chunking uses embeddings to split documents only when the actual topic shifts.

Why Does This Exist?

In a Retrieval-Augmented Generation (RAG) pipeline, you must split large documents into smaller pieces ("chunks") before embedding them. The most common approach is fixed-size chunking: splitting the text every 500 words, regardless of what the text says.

Fixed-size chunking is incredibly fast and easy to implement, but it creates a massive problem: it routinely slices right through the middle of cohesive thoughts. If a paragraph explains a complex chemical reaction, and the 500-word limit hits exactly halfway through the explanation, the two resulting chunks are mathematically orphaned. The first chunk has the setup, the second chunk has the conclusion, and neither chunk contains enough context to be retrieved accurately.

Semantic Chunking solves this by letting the meaning of the text dictate the boundaries. It groups sentences together as long as they remain on the same topic, and only creates a split when the subject matter genuinely shifts.

Think of It Like This

Slicing a mixed pizza

Imagine you order a large pizza that is 1/3 pepperoni, 1/3 mushroom, and 1/3 plain cheese.

Fixed-size chunking is like a machine that blindly drops a grid cutter over the pizza. You get perfectly identical square slices, but some slices are half pepperoni and half mushroom, ruining the flavor profile of both.

Semantic chunking is like a human carefully looking at the toppings and using a knife to slice exactly along the borders between the pepperoni, the mushroom, and the cheese. The slices might be slightly different shapes and sizes, but every single slice has a consistent, unified flavor.

How It Actually Works

Semantic chunking algorithms typically use a sliding window and an embedding model to evaluate the "distance" between consecutive sentences.

The Algorithm

Sentence Splitting: The entire document is parsed and split into individual sentences using standard NLP libraries (like NLTK or spaCy).
Sequential Embedding: Every individual sentence is converted into a vector embedding.
Similarity Calculation: The algorithm iterates through the list of sentences, calculating the cosine similarity between Sentence $N$ and Sentence $N+1$ .
Boundary Detection (Thresholding):
- If the similarity is high (e.g., 0.85), it means the two sentences are talking about the same concept. They are grouped into the same chunk.
- If the similarity drops below a predetermined threshold (e.g., 0.50), it signals a shift in topic. The algorithm draws a boundary here, finalizing the current chunk and starting a new one.

Enhancements

Comparing single sentences can be overly noisy (e.g., a short transitional sentence might artificially trigger a split). To smooth out the signal, production systems often compare groups of sentences. For example, it might compare a rolling window of [Sentence 1, 2, 3] against [Sentence 4, 5, 6].

Show Me the Code

This is a simplified demonstration of how to evaluate semantic boundaries between sentences using sentence-transformers and cosine similarity.

1from sentence_transformers import SentenceTransformer, util2import numpy as np3
4# 1. Initialize the embedding model5model = SentenceTransformer('all-MiniLM-L6-v2')6
7# 2. A document split into sentences. 8# Notice the clear topic shift at sentence index 3.9sentences = [10    "The central bank announced a new interest rate policy today.",11    "Officials stated this is an attempt to curb rising inflation.",12    "Markets reacted poorly, with the S&P 500 dropping 2%.",13    14    "In unrelated news, the Mars rover successfully drilled a new core sample.",15    "NASA scientists are excited to analyze the rock for signs of ancient water."16]17
18# 3. Embed all sentences19embeddings = model.encode(sentences)20
21# 4. Calculate similarities between consecutive sentences22similarities = []23for i in range(len(embeddings) - 1):24    sim = util.cos_sim(embeddings[i], embeddings[i+1]).item()25    similarities.append(sim)26
27# 5. Define a threshold to split the chunks28THRESHOLD = 0.2529chunks = []30current_chunk = [sentences[0]]31
32for i, sim in enumerate(similarities):33    print(f"Similarity between '{sentences[i][:15]}...' and '{sentences[i+1][:15]}...': {sim:.2f}")34    35    if sim >= THRESHOLD:36        # Keep grouping37        current_chunk.append(sentences[i+1])38    else:39        # Similarity dropped! Split here.40        chunks.append(" ".join(current_chunk))41        current_chunk = [sentences[i+1]]42
43# Add the final chunk44if current_chunk:45    chunks.append(" ".join(current_chunk))46
47print("\n--- Final Chunks ---")48for idx, chunk in enumerate(chunks):49    print(f"Chunk {idx}: {chunk}")50
51# -> Similarity between 'The central ban...' and 'Officials state...': 0.6552# -> Similarity between 'Officials state...' and 'Markets reacted...': 0.3553# -> Similarity between 'Markets reacted...' and 'In unrelated ne...': 0.04  <-- Sharp Drop!54# -> Similarity between 'In unrelated ne...' and 'NASA scientists...': 0.5255#56# -> --- Final Chunks ---57# -> Chunk 0: The central bank announced a new interest rate policy today. Officials stated this is an attempt to curb rising inflation. Markets reacted poorly, with the S&P 500 dropping 2%.58# -> Chunk 1: In unrelated news, the Mars rover successfully drilled a new core sample. NASA scientists are excited to analyze the rock for signs of ancient water.

Watch Out For

Massive ingestion latency

Semantic chunking requires running an embedding model on every single sentence (or small group of sentences) in your dataset before you even generate the final chunk embeddings. If you are indexing millions of pages, this can increase your ingestion compute time (and OpenAI API bill) by a factor of 10x compared to simple character-based splitting. You must weigh the improved retrieval accuracy against the much higher ingestion cost.

Runaway chunks

If a document consists of a slow, gradual narrative where no two consecutive sentences have a sharp drop in similarity, semantic chunking might group the entire 20-page document into a single chunk. You must always implement a fallback "maximum token length" to force a split if the semantic logic runs away, ensuring the chunk still fits inside the embedding model's context window.

The Quick Version

Fixed-size chunking randomly chops text based on character count, frequently severing the context of cohesive paragraphs.
Semantic chunking evaluates the mathematical meaning of the text to find logical boundaries.
It iterates through sentences, generating embeddings and calculating the cosine similarity between them.
When the similarity between adjacent sentences drops below a threshold, it signals a topic shift, and the algorithm draws a chunk boundary.