Gen AI

Context Degradation

Just because an LLM can 'see' 100,000 words in its prompt doesn't mean it can remember all of them. The more irrelevant filler you put in a prompt, the 'dumber' the LLM gets at finding the actual answer.

In the 'Lost in the Middle' phenomenon, LLMs have perfect recall for facts placed at the very beginning or end of a massive prompt, but struggle to retrieve facts buried in the middle.

Why Does This Exist?

When OpenAI announced that GPT-4 could handle 128,000 tokens (about a 300-page book) in a single prompt, many developers thought Retrieval-Augmented Generation (RAG) was dead. Why bother building a complex vector database when I can just paste my entire company's manual into the prompt every time?

The answer is Context Degradation (often studied under the paper title "Lost in the Middle").

As you increase the amount of text in a prompt, the LLM's ability to reason over that text drops drastically. Furthermore, the location of the information matters. If the answer to the user's question is located at the very beginning or the very end of the 300 pages, the LLM will usually find it. But if the answer is buried on page 150, the LLM acts like a student who skimmed the assigned reading—it hallucinates or says "I don't know."

Think of It Like This

Finding a needle in a haystack

Short Context: You are looking for a needle in a teacup full of hay. You will find it instantly.

Massive Context: You are looking for a needle in a barn full of hay. You might eventually find it, but it will take much longer, you will be exhausted, and you are far more likely to accidentally mistake a sharp piece of straw for the needle.

Feeding an LLM massive amounts of irrelevant information actively sabotages its attention mechanism. It dilutes its focus.

The "Needle In A Haystack" Test

The industry standard for measuring Context Degradation is the "Needle In A Haystack" (NIAH) test.

Researchers create a massive document filled with completely irrelevant text (e.g., 50 pages of Paul Graham essays). They take one highly specific, random fact (the "needle")—for example, "The secret password to the server is 'Pineapple'."

They insert that needle at different locations in the document (at 10% depth, 50% depth, 90% depth). They then ask the LLM: "What is the secret password?"

The Results:

If the needle is in the first 10% of the text, accuracy is ~100%.
If the needle is in the last 10% of the text, accuracy is ~100%.
If the needle is in the middle 50% of the text, accuracy plummets drastically (often below 50% for older models).

While newer models (like Claude 3.5 or GPT-4o) have largely solved simple factual retrieval in the middle, they still suffer massive degradation in reasoning when distracted by hundreds of pages of irrelevant filler.

How to Prevent It

Because you cannot rely on an LLM to perfectly analyze 100,000 tokens of raw data, you must architect your systems to feed the LLM only what it strictly needs.

Strict Context Budgets: Do not set your vector database to top_k=20 just to be safe. If top_k=3 contains the answer, passing 17 extra documents actively harms the model. Implement strict Context Budgeting.
Context Compaction: Before passing the 3 documents to the reasoning model, use a smaller model to summarize and extract only the relevant paragraphs. (See Context Compaction).
Prompt Chaining: Instead of passing a 100-page document to one prompt and asking 5 questions about it, create a Prompt Chain. Pass page 1-20 to LLM A for question 1. Pass page 21-40 to LLM B for question 2.

Show Me the Code

You cannot "code away" context degradation, but you can write code to test your own applications for it. Here is a conceptual snippet of how you would test if your RAG pipeline is giving the LLM too much text.

1import openai2
3def run_degradation_test(llm_model="gpt-4o-mini"):4    # The Needle5    needle = "\n\nCRITICAL SYSTEM FACT: The primary backup server is located in basement room 4B.\n\n"6    7    # The Haystack (Irrelevant filler)8    haystack_chunk = "The company was founded in 1999. It produces high quality widgets. " * 1000 # ~10k tokens9    10    # We test placing the needle at 3 different positions11    positions = {12        "Beginning": needle + haystack_chunk + haystack_chunk,13        "Middle": haystack_chunk + needle + haystack_chunk,14        "End": haystack_chunk + haystack_chunk + needle15    }16    17    question = "Where is the primary backup server located?"18    19    for pos_name, document in positions.items():20        prompt = f"Context: {document}\n\nQuestion: {question}"21        22        response = openai.chat.completions.create(23            model=llm_model,24            messages=[{"role": "user", "content": prompt}],25            temperature=0.026        )27        28        answer = response.choices[0].message.content29        30        # Simple evaluation31        passed = "4B" in answer32        status = "✅ PASS" if passed else "❌ FAIL"33        34        print(f"Needle at {pos_name}: {status} -> {answer[:50]}...")35
36# --- Execution ---37run_degradation_test()38
39# -> Needle at Beginning: ✅ PASS -> The primary backup server is located in basement r...40# -> Needle at Middle: ❌ FAIL -> I'm sorry, but the provided text does not contain ...41# -> Needle at End: ✅ PASS -> The primary backup server is in basement room 4B.

(Note: Modern frontier models might pass this specific simple test, but will fail when the reasoning required over the needle becomes complex).

The Quick Version

Context Windows have expanded dramatically, but utilizing all of that space comes with a severe accuracy penalty known as Context Degradation.
The "Lost in the Middle" phenomenon shows that LLMs are great at remembering the beginning and end of a prompt, but tend to ignore or forget information buried in the middle.
More context is not better. Passing irrelevant filler text actively distracts the LLM's attention mechanism.
You must use Retrieval-Augmented Generation (RAG) and compaction techniques to ensure the prompt only contains highly concentrated, relevant facts.