Chain-of-Thought Prompting

The Problem

Before 2022, large language models (LLMs) struggled significantly with multi-step reasoning tasks, such as math word problems, logic puzzles, or common-sense reasoning. The standard approach was "Standard Prompting," where a model was shown a few examples of a question and the final answer (few-shot learning), and then asked to predict the final answer for a new question.

Because Transformers generate text one token at a time, predicting the final answer directly means the model has to perform all the necessary logical leaps and mathematical calculations internally, in a single forward pass, before outputting the first token of the answer. For complex problems, this internal "jump" is simply too large, leading models to guess incorrectly. Scaling up the model size didn't fundamentally solve this bottleneck; even massive models failed at primary-school math.

The Idea

The Google Brain team realized that human reasoning doesn't work by instantly jumping to a final answer. Humans break complex problems into smaller, manageable steps.

The authors proposed Chain-of-Thought (CoT) Prompting. Instead of just showing the model the final answer in the few-shot examples, they provided a series of intermediate reasoning steps—a "chain of thought"—leading up to the answer. By simply changing the prompt format to include this step-by-step logic, the model learns to generate its own chain of thought for new problems. This forces the model to explicitly write out intermediate computations, giving it a textual "scratchpad" to work through the logic before committing to a final answer.

How It Works

The mechanism relies purely on in-context learning (prompting); no model weights are updated.

Standard Prompting:

• Prompt: Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: The answer is 11.
• New Question: Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?
• Model Output (Often wrong): A: The answer is 27. (Fails to compute correctly).

Chain-of-Thought Prompting:

• Prompt: Q: Roger has 5 tennis balls. He buys 2 more cans... A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11.
• New Question: Q: The cafeteria had 23 apples...
• Model Output (Correct): A: The cafeteria had 23 apples originally. They used 20 to make lunch. So they had 23 - 20 = 3. They bought 6 more apples, so they have 3 + 6 = 9. The answer is 9.

By outputting the intermediate steps, the model buys itself more "compute" (more forward passes through the network) to arrive at the correct answer. The attention mechanism can look back at the intermediate steps it just generated to help calculate the next step, rather than trying to do it all at once.

Why It Mattered

Chain-of-Thought prompting caused an immediate, massive leap in the performance of LLMs on reasoning benchmarks. On the GSM8K math word problem dataset, CoT prompting improved the performance of PaLM 540B from 17.9% to 58.1%, matching human baseline performance.

It proved that LLMs possessed latent reasoning capabilities that were being bottlenecked by the standard input-output format. It also revealed an "emergence" phenomenon: CoT only worked on sufficiently large models (typically >50B parameters); small models just produced illogical chains of thought. Furthermore, CoT made model outputs inherently interpretable. If the model got the answer wrong, developers could read the chain of thought to pinpoint exactly where the logic failed.

What Came After

CoT became the foundational technique for interacting with and evaluating LLMs. A follow-up paper famously showed that you didn't even need few-shot examples; simply appending the magic phrase "Let's think step by step" (Zero-Shot CoT) triggered the same reasoning behavior.

This paper launched an entire subfield of research into "reasoning-time compute." It directly inspired more complex search algorithms like Self-Consistency (generating many CoTs and taking the majority vote) and Tree of Thoughts. Today, advanced models like OpenAI's o1 and DeepSeek-R1 have internalized Chain-of-Thought, being trained via reinforcement learning to generate massive, hidden chains of thought automatically before returning an answer.