Gen AI

Parent-Child Chunking

Small chunks are easy to find via search, but bad for LLM context. Large chunks are hard to find via search, but great for LLM context. Parent-child chunking indexes the small chunks but retrieves the large ones.

The search engine finds the highly-specific 'child' chunk, but instead of sending just that sentence to the LLM, it traces back up the hierarchy and sends the entire 'parent' section.

Why Does This Exist?

In Retrieval-Augmented Generation (RAG), engineers face a frustrating paradox regarding chunk size.

Small chunks (e.g., 1-2 sentences): These are fantastic for the vector database. A short sentence creates a highly specific embedding. If a user asks a specific question, the small chunk will match perfectly. However, if you feed a single, isolated sentence into an LLM, the LLM lacks the surrounding context needed to write a comprehensive answer.
Large chunks (e.g., 5-10 paragraphs): These are fantastic for the LLM. It gets all the background information, caveats, and surrounding context. However, they are terrible for the vector database. A large chunk's embedding is an "average" of many different ideas. It is diluted. It won't match a highly specific user query.

Parent-Child Chunking (also called hierarchical chunking or the Auto-Merging Retriever) is an architecture pattern designed to give you the best of both worlds. You search across the small chunks to get high precision, but you retrieve the large chunk to get high context.

Think of It Like This

Using an index to read a textbook

Imagine a massive textbook on World War II.

If you are asked, "What date was the Normandy landing?", you don't read the whole book (a massive chunk). You go to the index at the back of the book, look for "Normandy," and find a highly specific pointer to "Page 412, Paragraph 2" (the child chunk).

The index helps you find the exact sentence immediately. But once you find that sentence, you don't just read that one sentence in isolation; you read the entire chapter (the parent chunk) so you understand the strategy, the weather conditions, and the aftermath of the landing.

The index (child chunks) provides precision. The chapter (parent chunk) provides context.

How It Actually Works

Implementing parent-child chunking requires changes to both the ingestion pipeline and the retrieval logic.

1. Hierarchical Ingestion

During the ingestion phase, documents are split twice. First, the document is split into Parent Chunks (e.g., 1,000 words each). Each parent chunk is assigned a unique ID (e.g., Parent-A) and stored in a standard database or document store (not necessarily a vector database).

Second, each parent chunk is further divided into smaller Child Chunks (e.g., 100 words each). These child chunks are embedded into vectors. Crucially, each child chunk's metadata contains a pointer to its parent's ID.

2. The Retrieval Phase

When a user submits a query:

Search: The query is embedded and compared against the vector database containing the child chunks. Because the child chunks are small and focused, the database easily finds the exact sentence or paragraph that matches the query.
Resolution (The Pivot): The system looks at the top result, sees that it is a child chunk, and reads its metadata pointer (e.g., "I belong to Parent-A").
Fetching: Instead of returning the child chunk to the LLM, the system queries the document store for Parent-A.
Generation: The entire 1,000-word parent chunk is injected into the LLM's prompt.

Advanced Concept: Auto-Merging (Thresholding)

If a user's query is broad, the vector search might return five different child chunks that all happen to belong to the exact same parent chunk.

Advanced implementations use an "Auto-Merging" threshold. For example, if a parent chunk has 10 children, and the search returns 3 of those children in the top results, the system says, "Clearly this entire parent section is highly relevant," and replaces the 3 children with the 1 parent. If the search only returns 1 child from a parent, the system might just return that single child to save context space.

Show Me the Code

Many orchestration frameworks like LlamaIndex have this pattern built-in, but understanding how to implement it manually clarifies the logic.

1# A simplified conceptual implementation of Parent-Child Retrieval2
3# 1. Our Document Store (Key-Value store for large Parent Chunks)4doc_store = {5    "parent_1": "The heart pumps blood through the circulatory system. "6                "It has four chambers: two atria and two ventricles. "7                "The right atrium receives deoxygenated blood from the body."8}9
10# 2. Our Vector DB (Stores small Child Chunks with pointers)11# In reality, this would be Pinecone or Qdrant storing vectors.12vector_db = [13    {"id": "child_1a", "text": "The heart pumps blood.", "parent_id": "parent_1"},14    {"id": "child_1b", "text": "It has four chambers.", "parent_id": "parent_1"},15    {"id": "child_1c", "text": "The right atrium receives deoxygenated blood.", "parent_id": "parent_1"}16]17
18def simulate_vector_search(query):19    # Simulating a vector search that perfectly matches a specific detail20    if "chambers" in query:21        return vector_db[1] # Returns child_1b22    return None23
24def retrieve_for_llm(query):25    # Step 1: Search the vector database (hits the small chunk)26    top_hit = simulate_vector_search(query)27    28    if top_hit:29        print(f"Vector search matched specific child: '{top_hit['text']}'")30        31        # Step 2: Read the pointer32        parent_id = top_hit["parent_id"]33        34        # Step 3: Fetch the parent chunk from the doc store35        parent_chunk = doc_store[parent_id]36        print(f"Retrieving parent chunk for LLM context: '{parent_chunk}'")37        38        return parent_chunk39    40    return "No results."41
42# Execute43query = "How many chambers does the heart have?"44context = retrieve_for_llm(query)45
46# -> Vector search matched specific child: 'It has four chambers.'47# -> Retrieving parent chunk for LLM context: 'The heart pumps blood through the circulatory system. It has four chambers: two atria and two ventricles. The right atrium receives deoxygenated blood from the body.'

Watch Out For

Context Window Bloat

Parent-child chunking fundamentally increases the amount of text sent to the LLM. If your standard vector search returns the top 5 results, and you swap all 5 of those small child chunks out for massive parent chunks, you might easily exceed your LLM's context window (or dramatically increase your API costs and latency). You must strictly limit how many parent chunks are fetched.

The Quick Version

Small chunks provide high search precision but poor LLM context. Large chunks provide poor search precision but great LLM context.
Parent-child chunking splits documents twice: into large parent blocks, and then into smaller child blocks.
The small child blocks are embedded and stored in the vector database.
When a search hits a child block, the system uses metadata pointers to retrieve the entire parent block and feed that to the LLM.