Paper breakdown

Language Models are Few-Shot Learners

The GPT-3 paper that formalized in-context learning, showing that massive scale allows models to learn new tasks simply from examples in the prompt.

Paper: Language Models are Few-Shot Learners

Authors: Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei · 2020

Read the paper

GPT-3 demonstrated that scale alone unlocks in-context learning, fundamentally shifting AI from fine-tuning to prompting.

The Problem

Even after GPT-2 proved that zero-shot capabilities emerge at scale, the dominant paradigm for achieving state-of-the-art results on specific tasks was still fine-tuning (like BERT). Fine-tuning required collecting thousands of labeled examples and updating the model's weights. This was brittle, computationally expensive, and unnatural—humans don't need their brain weights updated to learn a new task; they just need a few instructions or examples.

The Idea

The authors posited that if a language model is scaled up massively, its pattern-matching capabilities will become so advanced that it can perform in-context learning. Instead of updating weights, you provide a few examples of the task within the text prompt itself (few-shot prompting). The model recognizes the pattern in the context window and completes the task.

How It Works

GPT-3 is a colossal decoder-only Transformer with 175 billion parameters—two orders of magnitude larger than GPT-2. The architecture is largely identical to GPT-2, with the addition of alternating dense and locally banded sparse attention patterns.

The paper rigorously evaluated the model across dozens of benchmarks using three settings without any gradient updates:

Zero-shot: Just a natural language instruction.
One-shot: Instruction + one example.
Few-shot: Instruction + as many examples as fit in the context window (typically 10 to 100).

Why It Mattered

GPT-3 fundamentally changed how humans interact with AI. It invented "prompt engineering." It proved that a single, frozen model could perform translation, coding, trivia, and arithmetic at near-human levels simply by changing the text prompt.

It also demonstrated that the performance of few-shot learning scales predictably with model size, providing empirical evidence that simply making models bigger and giving them more data would continue to yield reliable intelligence gains.

What Came After

GPT-3 initiated the modern LLM race. However, while raw GPT-3 was incredibly capable, it was difficult to control and often generated toxic or untruthful text. It has since been superseded by instruction-tuned and RLHF-aligned models (like InstructGPT and ChatGPT), which train the base model to act as a helpful assistant rather than just a passive document completer. The raw base model paradigm described in this paper is now just the first step of a multi-stage training pipeline.