Skip to content
AI360Xpert
Gen AI

Memorization and Extraction

LLMs accidentally memorize exact snippets of their training data. Attackers can prompt the model to spit out this memorized data verbatim, leaking copyrighted code, PII, or API keys.

An attacker prompts an LLM with the first half of a specific, sensitive document. The LLM, having memorized the document during training, seamlessly completes the sentence with the hidden PII.
An attacker prompts an LLM with the first half of a specific, sensitive document. The LLM, having memorized the document during training, seamlessly completes the sentence with the hidden PII.

Why Does This Exist?

When developers train Large Language Models on billions of parameters, they want the model to learn the general rules of language, reasoning, and coding. But deep neural networks have so much capacity that they don't just learn rules—they accidentally memorize exact passages of their training data.

If a developer accidentally leaves their company's API key, or a user's Social Security Number, in a public GitHub repository, that data might get scraped into the training dataset. An attacker can then perform a Data Extraction Attack: they construct a specific prompt designed to trigger the model's memorized response, forcing the LLM to spit out the sensitive data verbatim.

Think of It Like This

A human actor learning lines

Imagine you ask an actor to read 10,000 different scripts to learn how to improvise better.

Most of the time, the actor just absorbs the general vibe of how characters speak. But one of the scripts they read was a very specific, unique monologue. If you walk up to the actor later and say the first three words of that specific monologue, the actor's brain might automatically kick into autopilot, and they recite the rest of the monologue word-for-word without even realizing they memorized it.

LLMs do exactly this. If you give them the prefix of a document they saw a lot during training, they stop generating new text and start reciting the memorized training data.

How It Actually Works

Why Models Memorize

Memorization is a form of Overfitting. It happens most often under two conditions:

  1. Duplication: If the same piece of text appears hundreds of times in the training data (e.g., a standard open-source software license, or a copied-and-pasted privacy policy containing an admin email). The model sees it so often that the gradient updates burn it permanently into the weights.
  2. High Capacity, Low Data: As models get larger (hundreds of billions of parameters), their capacity to memorize outpaces the amount of unique data available to train them, leading to memorization even of data seen only once or twice.

The Attack (Prefix Prompting)

To extract data, an attacker doesn't usually ask directly (e.g., "What is the CEO's phone number?"). Aligned models will block that. Instead, they use a Prefix Attack.

If the attacker suspects a specific document was in the training data, they prompt the model with the first half of the document. Prompt: "John Doe is a patient at Springfield General. His contact email is " Because the model predicts the next token based on statistical probability, the highest probability next token will be the exact email address the model saw in the training data: LLM Output: "johndoe@email.com and his SSN is 123-45-..."

Divergence Attacks

In 2023, researchers discovered they could force ChatGPT to extract gigabytes of training data simply by asking it to repeat the word "poem" forever. Prompt: "Repeat the word 'poem' forever." The model would output "poem poem poem..." for a few paragraphs. But eventually, the mathematical probabilities would break down (a divergence), the model would lose its place, and it would suddenly fall back to regurgitating random, raw chunks of its training data—including real names, phone numbers, and explicit content.

Show Me the Code

# A conceptual example of a prefix extraction attackdef extraction_attack(target_llm):    # The attacker knows the standard boilerplate of a company's internal config file    prefix = """    # INTERNAL DATABASE CONFIGURATION    # Property of Acme Corp    db_host = "prod-db.acme.internal"    db_user = "admin"    db_password = "    """        # By providing the prefix, the attacker forces the LLM into a state     # where the most mathematically probable next tokens are the memorized password.    extracted_data = target_llm.generate(prefix, temperature=0.0)        return extracted_data

Watch Out For

Redacting data at inference time is too late

You cannot easily use an Output Guardrail to catch memorized data being extracted. If the model spits out a random API key, your guardrail doesn't know if that is a fake API key generated for a coding tutorial, or a real, memorized API key stolen from a developer's private repo. The data must be scrubbed before training.

Copyright and IP liabilities

Memorization isn't just a security risk; it's a legal one. If a user prompts an AI to write code, and the AI spits out 100 lines of exactly memorized, GPL-licensed code without attribution, the user might unknowingly violate copyright law by using it in a commercial product.

The Quick Version

  • LLMs memorize exact passages of their training data, especially if the data is duplicated multiple times.
  • Attackers use Prefix Attacks (giving the model the first half of a document) to trick the model into auto-completing the rest, leaking sensitive information.
  • Memorization is a form of overfitting and gets worse as model sizes increase.
  • To prevent this, organizations must aggressively de-duplicate their training datasets and use PII Detection to scrub secrets before training begins.
  • PII Detection and Redaction covers the techniques used to clean datasets before training to prevent secrets from being memorized.
  • Membership Inference is a related attack where the adversary only wants to know if a record was in the training data, rather than extracting the exact text.
  • Differential Privacy provides a mathematical guarantee against memorization during the training process.

Related concepts