Gen AI

Self-Consistency

Instead of asking the LLM to think through a problem once, you ask it to think through the problem 5 different times in parallel. You then look at the 5 final answers and pick the one that appeared most often.

Why Does This Exist?

Chain of Thought (CoT) prompting is incredibly powerful for solving math and logic problems. It forces the LLM to write out its reasoning step-by-step.

However, LLMs are probabilistic engines. If you ask an LLM the exact same math problem three times, it might write out three different chains of reasoning.

Attempt 1: Perfect logic $\rightarrow$ Answer is 42.
Attempt 2: Perfect logic $\rightarrow$ Answer is 42.
Attempt 3: Makes a dumb arithmetic mistake in step 2 $\rightarrow$ Answer is 17.

If your application only runs the prompt once, and it happens to generate Attempt 3, your user gets the wrong answer.

Self-Consistency is an ensemble technique that embraces this probabilistic nature. It intentionally runs the exact same CoT prompt multiple times, generates multiple diverse reasoning paths, and then uses a simple "majority vote" to select the final answer.

Think of It Like This

A panel of experts

Standard CoT: You ask one accountant to calculate your taxes. They work through the math on a whiteboard. They make a small addition error on line 4, resulting in a final tax bill of $5,000. You pay it, unaware it was wrong.

Self-Consistency: You hire 5 identical accountants. You put them in 5 separate rooms and ask them to calculate your taxes.

Accountant 1: $4,500
Accountant 2: $4,500
Accountant 3: $5,000 (made an addition error)
Accountant 4: $4,500
Accountant 5: $4,500

You look at the results. Because 4 out of 5 accountants arrived at $4,500 using different scratchpads, you can be extremely confident that$ 4,500 is the correct answer. You ignore the outlier.

How It Actually Works

Implementing Self-Consistency requires two key components: Temperature and Parsing.

1. High Temperature Generation

Normally, for math and logic tasks, developers set the LLM's temperature to 0.0. This forces the model to be completely deterministic, always picking the most likely next word.

However, if temperature is 0, running the prompt 5 times will result in the exact same 5 reasoning paths. Self-Consistency requires diversity in reasoning. Therefore, you must set the temperature slightly higher (e.g., 0.5 or 0.7). This encourages the model to try different logical approaches or phrasing in each parallel run.

2. The Majority Vote (Parsing)

Because each run will output a long paragraph of reasoning, you cannot just do a simple string comparison of the whole output. You must parse out the final answer from each run.

If the prompt forces the LLM to end with Final Answer: [X], your code extracts [X] from all 5 runs. You then count which [X] appears most frequently (the mode).

Show Me the Code

This code demonstrates how to execute 5 parallel CoT calls and calculate the majority vote.

1import openai2import concurrent.futures3from collections import Counter4
5def run_single_cot(prompt):6    # Notice we use a non-zero temperature to ensure diverse reasoning paths7    response = openai.chat.completions.create(8        model="gpt-4o-mini",9        messages=[{"role": "user", "content": prompt}],10        temperature=0.7 11    )12    return response.choices[0].message.content13
14def self_consistency_solve(problem, num_paths=5):15    prompt = f"""16    Solve the following problem step by step.17    End your response with EXACTLY: 'Final Answer: [Number]'18    19    Problem: {problem}20    """21    22    # 1. Run multiple paths in parallel23    print(f"Running {num_paths} parallel reasoning paths...")24    with concurrent.futures.ThreadPoolExecutor() as executor:25        results = list(executor.map(lambda _: run_single_cot(prompt), range(num_paths)))26        27    # 2. Parse out the final answers28    extracted_answers = []29    for i, res in enumerate(results):30        try:31            # Extract whatever comes after "Final Answer:"32            answer = res.split("Final Answer:")[-1].strip()33            extracted_answers.append(answer)34            print(f"Path {i+1} concluded: {answer}")35        except IndexError:36            print(f"Path {i+1} failed to format correctly.")37            38    # 3. Majority Vote39    vote_counts = Counter(extracted_answers)40    winning_answer, votes = vote_counts.most_common(1)[0]41    42    print("\n--- Majority Vote Result ---")43    print(f"Winning Answer: {winning_answer} ({votes}/{num_paths} votes)")44    return winning_answer45
46# --- Execution ---47complex_math = "If I have 32 apples, give half to Mary, buy 10 more, and then split the total evenly with John, how many apples do I have left?"48self_consistency_solve(complex_math)49
50# -> Running 5 parallel reasoning paths...51# -> Path 1 concluded: 1352# -> Path 2 concluded: 1353# -> Path 3 concluded: 26  (Hallucinated/Forgot the final split)54# -> Path 4 concluded: 1355# -> Path 5 concluded: 1356# -> 57# -> --- Majority Vote Result ---58# -> Winning Answer: 13 (4/5 votes)

Watch Out For

5x Token Cost

Self-Consistency is a brute-force approach to accuracy. If you run the prompt 5 times, you are paying 5 times as much in API fees and waiting 5 times as long (if not parallelized). You should only use Self-Consistency for extremely complex reasoning tasks where a hallucination is catastrophic (like legal analysis, complex math, or medical triage). Do not use it for simple summarization.

The Quick Version

Standard Chain of Thought (CoT) runs a single reasoning path. If the LLM makes a mistake early in the path, the final answer is wrong.
Self-Consistency runs the exact same CoT prompt multiple times in parallel, using a non-zero temperature to force diverse reasoning paths.
The system extracts the final answer from every path and takes a majority vote.
The majority vote acts as a mathematical buffer against random hallucinations, drastically increasing the reliability of the system.