Gen AI

RAG Failure Diagnosis

A bad answer has four possible origins — ingestion, chunking, retrieval, generation — and they get checked in that order, cheapest first, instead of jumping straight to prompt edits.

A bad answer is diagnosed cheapest-first: check whether the right chunk is even in the index, then whether retrieval returned it, then whether the model actually used it — skipping straight to prompt edits is the most common wasted week

Why Does This Exist?

RAG evaluation established that retrieval quality and generation quality are two separate scores, worth measuring independently — but even with that split in place, a single wrong answer still needs to be traced back to its actual cause before anyone can fix it. And RAG architecture has genuinely more failure points than just "retrieval versus generation": a document can fail to be properly ingested in the first place, a chunk boundary can split the needed information across two pieces, retrieval can fail to surface a chunk that's indexed correctly, or the model can ignore a chunk it was correctly handed.

The instinctive response to a wrong answer — edit the prompt — only ever fixes the last of those four possible causes. Applied to any of the other three, it changes nothing about the actual bug and burns real time on a fix that was never going to work, which is exactly the "standard wasted week" this page's diagram calls out directly.

Think of It Like This

Tracing a broken delivery back through the supply chain

A customer never received a package they ordered, and there are several genuinely different places the failure could have happened: maybe the item was never actually stocked in the warehouse, maybe it was stocked but misplaced on the wrong shelf, maybe warehouse staff picked the wrong item entirely, or maybe the correct item was picked and shipped but the customer never opened the box. Calling the customer to apologize more sincerely, or rewriting the "your order has shipped" email, does nothing to fix any of the first three causes — it only possibly helps if the failure genuinely happened at the very last step.

A competent investigation checks the warehouse's actual inventory first (is the item even in stock, at all), then the shelving records (was it findable), then the picking log (was the right item actually grabbed), and only then looks at delivery and communication. Diagnosing a RAG failure follows exactly this same cheapest-first, upstream-first order.

How It Actually Works

Four possible origins, checked in a specific order

The diagram lays the sequence out directly, and the ordering is deliberate: each check is strictly cheaper and faster than the one after it, so ruling out cheap explanations first avoids wasting time on an expensive one that was never the actual cause.

Ingestion — is the information even in the index at all? Check first by searching the raw indexed content directly, bypassing retrieval and generation entirely. If it's not there, nothing downstream can ever produce a correct answer.

Chunking — assuming the source was ingested, did chunking preserve it as a coherent unit, or did a boundary split it across two pieces, with neither half complete? Check by looking directly at the stored chunks near where the answer should be.

Retrieval — assuming an intact chunk exists, did search actually return it for this query? Check by running the query against the index directly, independent of what the model then did with the result.

Generation — assuming the correct chunk was retrieved and handed to the model, did the model actually use it? This is the only step where a prompt change can be the right fix, and it's checked last, not first.

Why cheapest-first matters in practice

Checking ingestion is often a direct text search over the raw indexed content — fast, and immediately conclusive if the information isn't there. Checking generation quality properly requires running the full pipeline and judging whether the output was faithful to what it received — meaningfully more expensive. Starting cheap and only escalating once cheaper explanations are ruled out is what keeps a session from burning time on generation quality when the real bug was a document that was never indexed.

What each stage's symptom actually looks like

An ingestion failure produces an answer missing information no version of the system could have supplied. A chunking failure often produces a partially correct answer, since half the needed information exists in whichever chunk was retrieved. A retrieval failure produces an answer missing information that genuinely exists in the index, if only it had been found. A generation failure produces an answer that contradicts material verifiably sitting in the context the model was handed — the one symptom a prompt fix can actually address.

Show Me the Code

A minimal diagnostic script that walks the four stages in order, stopping at whichever stage first reveals the actual problem.

1def diagnose(query: str, expected_answer_fragment: str, raw_documents: list[str],2             chunks: list[str], retrieved_chunks: list[str], generated_answer: str) -> str:3    if not any(expected_answer_fragment in doc for doc in raw_documents):4        return "INGESTION: the answer was never in the source documents at all"5    if not any(expected_answer_fragment in chunk for chunk in chunks):6        return "CHUNKING: the answer exists in source docs but no single chunk contains it whole"7    if not any(expected_answer_fragment in chunk for chunk in retrieved_chunks):8        return "RETRIEVAL: the right chunk exists but search did not return it for this query"9    if expected_answer_fragment not in generated_answer:10        return "GENERATION: the model had the right chunk and still produced a wrong answer"11    return "No failure detected at any stage"12
13
14result = diagnose(15    query="what is the refund window",16    expected_answer_fragment="30 days",17    raw_documents=["Our refund policy allows returns within 30 days of purchase."],18    chunks=["Our refund policy allows returns within 30 days of purchase."],19    retrieved_chunks=["Our shipping policy covers most regions."],   # wrong chunk retrieved20    generated_answer="I don't see information about the refund window.",21)22print(result)  # -> RETRIEVAL: the right chunk exists but search did not return it for this query

The function checks each stage strictly in order and returns immediately at the first one that fails — exactly the discipline that keeps a real debugging session from investigating generation quality when the actual bug sits three stages upstream.

Watch Out For

Jumping straight to prompt edits without checking the upstream stages

Editing the prompt in response to a wrong answer, without first confirming the correct chunk was even retrieved, is the single most common wasted effort in RAG debugging. If the actual cause is a retrieval or ingestion failure, no prompt wording fixes it, and the time spent iterating on prompt phrasing produces no improvement — while the actual bug remains completely unaddressed and undiagnosed.

Assuming one bad answer means one specific stage is broken systemically

A single wrong answer traced to, say, a retrieval failure doesn't necessarily mean retrieval is broken for every query — it might be a failure specific to that one query's phrasing or that one document's chunking. Treating every diagnosis as evidence of a system-wide problem in that stage, rather than checking whether the same failure recurs across a representative sample of queries, can lead to over-correcting one stage while missing that the actual failure rate there is low.

The Quick Version

A bad RAG answer can originate at ingestion, chunking, retrieval, or generation — four genuinely different failure points.
Diagnosis proceeds cheapest-first: check whether the answer is even indexed, then whether chunking preserved it, then whether retrieval found it, then whether generation used it.
Prompt edits can only ever fix a generation-stage failure — checked last, not first, and only actually useful if that's where the problem really is.
Each stage's failure produces a distinguishable symptom, from a completely missing answer to a confidently wrong one that ignores correct retrieved context.
Skipping straight to prompt tuning without checking the cheaper upstream stages is the standard, avoidable wasted debugging effort in RAG systems.