Self-Consistency

The Problem

Chain-of-Thought (CoT) prompting was a massive leap forward for LLM reasoning, allowing models to solve complex math and logic problems by generating step-by-step scratchpads. However, the standard way models generate text is through "greedy decoding"—picking the single most likely next word at every step.

This meant the model was locked into a single reasoning path. If the model made a tiny arithmetic error or logical slip early in the chain, the rest of the chain would be derailed, leading to a wrong final answer. Human reasoning, by contrast, is often exploratory: if we get stuck or reach an absurd conclusion, we try a different approach. Standard CoT was brittle because it placed all its bets on a single, unverified train of thought.

The Idea

The authors introduced Self-Consistency, a wildly simple but highly effective decoding strategy that treats reasoning as a probabilistic search rather than a deterministic sequence.

Instead of generating one chain of thought using greedy decoding, they proposed using temperature sampling to generate many diverse chains of thought for the exact same prompt. Because the model is sampling probabilistically, it will explore different ways of breaking down the problem. Some of these paths will contain mistakes. But crucially, the authors hypothesized that for problems with a fixed answer (like math), the correct reasoning paths are more likely to converge on the same final answer, while incorrect paths will result in a scattered variety of wrong answers.

Therefore, by simply looking at the final answers produced by all the different chains and taking a majority vote, you can dramatically increase the reliability of the model.

How It Works

The self-consistency method operates purely at inference time and requires no changes to the model weights or the prompt itself.

1. Prompting: You provide the model with a standard Chain-of-Thought prompt (e.g., a few-shot prompt with step-by-step examples).
2. Sampling: Instead of asking the model for its best guess (greedy decoding), you set the temperature slightly higher than zero and ask the model to generate $N$ different responses (e.g., $N=40$).
3. Extraction: You parse the final answer out of each of the $N$ generated chains of thought.
4. Marginalization (Voting): You find the most frequent final answer among the set. This is the "self-consistent" answer.

For example, if you sample 5 paths:

• Path 1: (Good logic) -> Final Answer: 18
• Path 2: (Arithmetic error) -> Final Answer: 24
• Path 3: (Alternative good logic) -> Final Answer: 18
• Path 4: (Misread the question) -> Final Answer: 12
• Path 5: (Good logic) -> Final Answer: 18

The majority vote is 18. The model outputs 18 as its final, high-confidence answer, effectively filtering out the hallucinations and calculation errors in Paths 2 and 4.

Why It Mattered

Self-Consistency yielded massive performance gains over standard Chain-of-Thought on virtually every reasoning benchmark, often pushing scores up by 10-15 absolute percentage points. It worked consistently across different models (PaLM, GPT-3, LaMDA) and tasks.

It demonstrated a profound property of large language models: they often "know" the right answer, but a single greedy path might not reflect their true internal confidence. By sampling, we can marginalize out the noise of brittle text generation. It also proved that throwing more inference-time compute (generating 40 paths instead of 1) directly translates to better reasoning performance, a concept that is foundational to modern AI scaling laws.

What Came After

Self-consistency became the absolute baseline for how LLMs are evaluated on reasoning tasks. Any modern benchmark score reported on datasets like GSM8K or MATH usually uses "CoT + Self-Consistency (maj@8)" or something similar.

It also paved the way for more sophisticated search algorithms. While Self-Consistency searches by blindly sampling entire paths and voting at the end, later methods like Tree of Thoughts (ToT) evaluate intermediate steps during generation, allowing the model to backtrack or prune bad reasoning paths before finishing the sentence. However, because of its extreme simplicity and effectiveness, Self-Consistency remains one of the most widely used inference techniques in production AI today.