Gen AI

Chunking Strategies

Splitting a document at fixed intervals can cut a sentence in half, while splitting along the document's own structure keeps each idea whole — the choice trades retrieval precision against surviving context.

Fixed-size chunking cuts a document at even intervals regardless of meaning, while structure-aware chunking cuts along the document's own section boundaries, keeping each idea whole inside one chunk

Why Does This Exist?

Embeddings work best on passages that are focused and specific, not on entire documents — embedding a 50-page manual as a single vector produces a representation so averaged across every topic the document touches that it's useless for finding the one specific paragraph a query actually needs. So real retrieval systems split documents into smaller pieces, called chunks, before embedding each one separately.

That split isn't a mechanical afterthought. Where a chunk boundary falls determines whether a retrieved passage actually contains the complete idea a query needs, or whether it's been sliced in half — a sentence broken across two separate chunks, with each half missing the context the other half provided. A chunking strategy that ignores document structure entirely is the single most common, and most avoidable, source of retrieval quality problems in a real RAG system.

Think of It Like This

Cutting a cake by the ruler versus cutting along the layers

Cutting a layered cake into equal-sized pieces with a ruler, ignoring where the layers and fillings actually sit, produces pieces that are consistently sized but sometimes slice straight through a filling in an awkward, half-there way — one piece gets most of the filling, the neighboring piece gets almost none, even though both looked identical by the ruler's measurement.

Cutting along the cake's actual structure — following where one layer ends and gives way to filling, then the next layer — produces pieces of varying size, but each one is a complete, coherent unit: filling and cake together, never split mid-bite. Fixed-size chunking is the ruler. Structure-aware chunking follows the cake's own layers.

How It Actually Works

The simplest approach splits a document into pieces of a fixed token or character count, optionally with some overlap between consecutive chunks so information near a boundary survives in at least one of the two chunks it falls between. This is easy to implement and works reasonably well on unstructured, uniform text, but it has no awareness of sentence or paragraph boundaries — the diagram above shows exactly this failure, a chunk boundary landing mid-sentence, splitting one idea across two separate, independently-embedded pieces.

Structure-aware chunking: respecting the document's own boundaries

A better approach splits along the document's actual structure — paragraph breaks, section headers, list boundaries — so each resulting chunk contains one complete, coherent unit rather than an arbitrary slice. This usually means chunks end up varying in size rather than being uniform, but each one is far more likely to be independently meaningful, which matters enormously once a chunk is retrieved on its own and handed to a model with no other context around it. Recursive splitting — trying to split at the largest structural boundary first (section, then paragraph, then sentence) and only falling back to a smaller unit when a piece is still too large — is a common practical implementation of this idea.

The size tradeoff that never fully goes away

However chunks are drawn, their size is a genuine tradeoff, not a setting with one correct answer. Smaller chunks retrieve more precisely — a very specific query is more likely to match a small, focused chunk directly on target — but each chunk carries less surrounding context, so a model reading it in isolation may lack information that was sitting just outside the chunk's boundary. Larger chunks carry more context per retrieved piece, but dilute the embedding's specificity, making it harder for a precise query to distinguish one large chunk from another that happens to also mention the query's topic in passing. There's no universally correct chunk size; it's a decision tuned against the specific documents and query patterns a system actually needs to serve.

What overlap actually buys

Adding overlap between consecutive chunks — repeating the last portion of one chunk as the start of the next — is a direct, low-cost mitigation for the fixed-size chunking failure mode above: information sitting near a boundary has a second chance to appear fully within at least one chunk, rather than being split with neither half containing the complete thought. Overlap doesn't fix the more fundamental structure-blindness problem, but it meaningfully reduces how often a boundary actually destroys a specific piece of information.

Show Me the Code

Fixed-size chunking with overlap, and a simple structure-aware alternative that splits on paragraph breaks, run against the identical text to show the difference in output.

1def fixed_size_chunks(text: str, size: int, overlap: int) -> list[str]:2    chunks, start = [], 03    while start < len(text):4        chunks.append(text[start:start + size])5        start += size - overlap                 # step forward by less than `size`, creating overlap6    return chunks7
8
9def paragraph_chunks(text: str) -> list[str]:10    return [p.strip() for p in text.split("\n\n") if p.strip()]11
12
13text = "First idea, stated fully here.\n\nSecond idea, also stated fully, in its own paragraph."14print(fixed_size_chunks(text, size=40, overlap=10))15# -> ['First idea, stated fully here.\n\nSeco', 'y here.\n\nSecond idea, also stated fully', 'ted fully, in its own paragraph.']16print(paragraph_chunks(text))17# -> ['First idea, stated fully here.', 'Second idea, also stated fully, in its own paragraph.']

The fixed-size version cuts straight through the boundary between the two ideas, landing part of each idea in an awkward middle chunk — the paragraph-aware version keeps each idea intact as its own chunk, exactly matching the document's own structure.

Watch Out For

Picking a chunk size once and never revisiting it

Chunk size interacts with the specific documents and query patterns a system serves, and a size that works well for one kind of content (short FAQ entries) can perform poorly for another (long technical procedures with many interdependent steps) inside the same system. Treating chunk size as a one-time setup decision, rather than something to measure and tune against real retrieval quality on the actual corpus, leaves an easy, low-cost improvement unclaimed.

Ignoring structure because fixed-size chunking is easier to implement

Fixed-size chunking is genuinely simpler to write and reason about, which makes it a tempting default even for well-structured documents — technical manuals, legal contracts, documentation with clear headers — where the document itself is practically handing over exactly where the meaningful boundaries are. Ignoring that available structure in favor of uniform-size cuts throws away information that would have directly improved chunk coherence at essentially no extra engineering cost.

The Quick Version

Chunking splits a document into smaller pieces before embedding, since embedding an entire document produces a representation too averaged to be useful.
Fixed-size chunking is simple but can cut a sentence or idea in half, ignoring the document's own structure.
Structure-aware chunking splits along paragraph, section, or list boundaries, keeping each chunk a coherent unit at the cost of uniform size.
Chunk size trades retrieval precision (favoring smaller chunks) against surviving context (favoring larger chunks) — there's no universally correct size.
Overlap between consecutive chunks mitigates, but doesn't eliminate, the boundary-splitting problem fixed-size chunking creates.