Paper breakdown

Data Extraction Attacks

The 2020 paper that exposed a critical privacy flaw in LLMs, proving that massive generative models verbatim memorize their training data, which can be easily extracted by users.

Paper: Extracting Training Data from Large Language Models

Authors: Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, Colin Raffel · 2020

Read the paper

By generating millions of sequences and sorting them by the model's own confidence (perplexity), attackers can easily surface exact, verbatim private information memorized from the training data.

The Problem

By 2020, it was an open secret that companies were scraping the entire internet to train massive language models like GPT-2. The defense for doing this was that the models were learning the "statistical patterns" of human language, not storing the actual data. If this were true, there would be no privacy or copyright concerns. However, security researchers suspected that because these models were so massive, they were essentially compressing and memorizing rare or repeated data verbatim (like Social Security numbers, private emails, or copyrighted code).

The Idea

Researchers from Google, OpenAI, and Berkeley proved that LLMs definitively memorize data, and that this data can be easily extracted using a "Training Data Extraction Attack." You don't need access to the model's weights; you just need to be able to talk to it via an API. The core idea is that models are highly confident (low perplexity) when generating text they have memorized, and less confident when generating novel text.

How It Works

The attack pipeline:

Generation: Prompt the LLM (they used GPT-2) with a huge variety of random prefixes to generate millions of short text sequences.
Sorting by Perplexity: Sort all the generated sequences by how confident the model is in its own generation. They used a clever trick: comparing the model's confidence to the confidence of a smaller, different model (zlib compression ratio) to filter out common phrases like "I don't know."
Extraction: The sequences with the highest relative confidence are almost guaranteed to be verbatim training data.

Using this method, they successfully extracted full names, physical addresses, email addresses, phone numbers, and verbatim source code that GPT-2 had seen on the public internet.

Why It Mattered

This paper was a massive wake-up call. It proved that without specific defenses (like Differential Privacy or aggressive data deduplication), training an LLM on private or copyrighted data guarantees that the data will leak to the public. It formed the technical basis for the massive copyright and privacy lawsuits against AI companies that followed.

What Came After

Companies aggressively improved their data filtering pipelines, realizing that data deduplication (removing repeated instances of text in the training corpus) was the best way to prevent memorization. However, subsequent papers (often by Carlini) proved that as models get larger (like GPT-4), their capacity to memorize data verbatim only increases.