Gen AI

Chain of Thought

Instead of forcing the LLM to output the final answer immediately, you prompt it to 'think step by step'. By outputting the intermediate steps, the model essentially creates a scratchpad for itself, drastically improving math and logic performance.

Why Does This Exist?

Large Language Models (LLMs) do not "think" the way humans do; they predict the next mathematical token in a sequence.

If you ask an LLM a complex math word problem, and the very next token it generates is the final number, the model has a very high chance of getting it wrong. Why? Because generating the final number immediately requires doing all the intermediate math in a single forward pass of the neural network.

Chain of Thought (CoT) prompting is a technique that forces the LLM to output the intermediate steps of its reasoning before outputting the final answer.

Because LLMs can "see" the text they just generated, outputting step 1 helps the model accurately predict step 2. Outputting step 2 helps it accurately predict step 3. The generated text acts as a cognitive scratchpad.

Think of It Like This

Doing math in your head vs. on paper

Imagine someone asks you: "What is 14 times 16?"

Standard Prompting: You are forced to shout the final answer out loud instantly, within one second, without moving your lips. You will probably guess wrong.

Chain of Thought: You are given a whiteboard. You write down:

$14 \times 10 = 140$
$14 \times 6 = (10 \times 6) + (4 \times 6) = 60 + 24 = 84$
$140 + 84 = 224$

You read the whiteboard, and confidently declare the final answer is 224. Chain of Thought gives the LLM that same whiteboard.

How It Actually Works

There are two primary ways to implement Chain of Thought prompting: Zero-Shot CoT and Few-Shot CoT.

1. Zero-Shot CoT ("Let's think step by step")

This is the famous magic phrase discovered in 2022. You append a single sentence to the end of your prompt: "Let's think step by step."

Prompt: "Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? Let's think step by step." LLM Output: "Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11."

Just adding that one sentence increased accuracy on the GSM8K math benchmark from 17% to 78% on early models.

2. Few-Shot CoT

Zero-Shot CoT relies on the model figuring out how to break down the steps. In more complex domain-specific tasks, the model might break it down wrong. Few-Shot CoT solves this by providing explicit examples of the reasoning process in the prompt.

Prompt Template:

1Question: [Sample Question]2Answer: [Step 1 logic]. [Step 2 logic]. [Step 3 logic]. The final answer is [X].3
4Question: [Actual Question]5Answer:

By showing the model exactly how a human expert reasons through a problem, the model will adopt that specific analytical framework for the real question.

Show Me the Code

Implementing Few-Shot CoT in a Python script is straightforward. We use the <think> or <scratchpad> XML tags to keep the reasoning organized, making it easy to parse out the final answer later.

1import openai2import re3
4def chain_of_thought_solve(math_problem):5    # We provide one example of the problem, complete with the internal reasoning steps6    prompt = f"""7    Solve the math problem. You must show your work inside <think> tags before providing the final numerical answer.8    9    Question: The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?10    <think>11    1. Start with 23 apples.12    2. Used 20: 23 - 20 = 3 apples left.13    3. Bought 6 more: 3 + 6 = 9 apples.14    </think>15    Final Answer: 916    17    Question: {math_problem}18    """19    20    response = openai.chat.completions.create(21        model="gpt-4o-mini",22        messages=[{"role": "user", "content": prompt}],23        temperature=0.024    )25    26    raw_output = response.choices[0].message.content27    28    # We can parse the output to hide the reasoning from the user if we want29    reasoning = re.search(r'<think>(.*?)</think>', raw_output, re.DOTALL)30    final_answer = raw_output.split("Final Answer:")[-1].strip()31    32    print("--- Behind the Scenes (LLM Scratchpad) ---")33    if reasoning:34        print(reasoning.group(1).strip())35        36    print("\n--- Final User-Facing Answer ---")37    print(final_answer)38
39# --- Execution ---40problem = "A juggler can juggle 16 balls. Half of the balls are golf balls, and half of the golf balls are blue. How many blue golf balls are there?"41chain_of_thought_solve(problem)42
43# -> --- Behind the Scenes (LLM Scratchpad) ---44# -> 1. Total balls = 16.45# -> 2. Half are golf balls: 16 / 2 = 8 golf balls.46# -> 3. Half of the golf balls are blue: 8 / 2 = 4 blue golf balls.47# -> 48# -> --- Final User-Facing Answer ---49# -> 4

Watch Out For

Increased Token Costs and Latency

Chain of Thought requires the model to output significantly more tokens. If a direct answer is 5 tokens, a CoT answer might be 150 tokens. Because LLMs generate text sequentially, generating 150 tokens will take 30 times longer, and you will be billed for all of those generated tokens. You should only use CoT for complex reasoning tasks, not for simple fact retrieval (like "What is the capital of France?").

The Quick Version

LLMs struggle with complex math and logic if they are forced to output the final answer instantly.
Chain of Thought (CoT) prompting forces the model to generate intermediate reasoning steps first (the "scratchpad").
The generated steps act as context for the model, vastly improving the accuracy of the final answer.
Zero-Shot CoT uses phrases like "Let's think step by step."
Few-Shot CoT provides explicit examples of the reasoning process to guide the model.