The Complete Guide to LLM Chain-of-Thought (CoT): From First Principles to Prompt Engineering Best Practices

1. The Essence of CoT in One Sentence

CoT gives the model a sheet of scratch paper—it transforms the model’s implicit internal reasoning into explicit external reasoning.

An LLM generates autoregressively, one token at a time. The probability of each new token depends on all previously generated tokens. CoT uses this mechanism by having the model first “write down” intermediate reasoning steps as tokens. Those tokens remain in the context and become input for subsequent generation, effectively providing the model with “external working memory.”

Autoregressive means that the model generates only one token at a time, and the generation of every new token depends on all tokens generated before it.

“Auto” means “self,” while “regressive” means “regression/dependence.” Together, the term means that the model depends on its own previous output.

For an intuitive analogy, imagine writing an article one character at a time. When you write the first character, you have only the title in mind. Before writing the second, you look at the first and decide what comes next. Before writing the third, you look at the first two, and so on. You never write the entire article all at once. At each step, you decide “what to write next” based on “what has already been written.”

An LLM does exactly the same thing, except that its unit of writing is a token rather than a character.

In precise mathematical terms, generating a complete sequence means calculating a chain of conditional probabilities:

P(entire output) = P(token₁) × P(token₂|token₁) × P(token₃|token₁,token₂) × ...

Expanding each step:

Step 1: The model sees [prompt]
        → calculates P(token₁ | prompt)
        → samples token₁

Step 2: The model sees [prompt, token₁]
        → calculates P(token₂ | prompt, token₁)
        → samples token₂

Step 3: The model sees [prompt, token₁, token₂]
        → calculates P(token₃ | prompt, token₁, token₂)
        → samples token₃

...the loop continues until the end-of-sequence token <EOS> is generated

Notice the key point: every step is one complete Transformer forward pass. The model sends all currently known tokens—the prompt plus generated tokens—through all N Transformer layers, obtains a probability distribution for the next token, and samples one token from that distribution.

Consider a concrete example.

Suppose the prompt is “The capital of France is”. The model generates as follows:

Input: [The, capital, of, France, is]
    → Transformer forward pass → probability distribution for the next token:
      "Par" (0.92), "the" (0.03), "a" (0.01), ...
    → sample → "Par"

Input: [The, capital, of, France, is, Par]
    → Transformer forward pass → probability distribution for the next token:
      "is" (0.97), "ty" (0.01), ...
    → sample → "is"

Input: [The, capital, of, France, is, Par, is]
    → Transformer forward pass → probability distribution for the next token:
      "." (0.85), "," (0.05), ...
    → sample → "."

Once “Par” is generated, the probability of “is” rises sharply to 0.97 because, in the training data, “Par” is almost always followed by “is.” This is the core of autoregressive generation: earlier output strongly constrains the possibilities that follow.

Why is this concept so important?

Because every prompt engineering technique discussed here ultimately manipulates this step-by-step generation process.

2. Why CoT Works: The Underlying Principles

2.1 The Fixed-Depth Limitation of Transformers

The number of Transformer layers, N, is a fixed architectural constant. No matter how difficult the question is, the model runs only these N layers before producing a result. Each layer gathers information through attention and processes it through an FFN, so the total computational depth is fixed at N.

When a task requires more reasoning steps than the model can complete within N layers, the model runs out of capacity. This is why LLMs are accurate at simple arithmetic but often fail at multiplication with many digits.

2.2 How CoT Breaks Through This Limitation

Without CoT, reasoning depth is fixed at N layers—one forward pass. With CoT, every generated intermediate result becomes input context for the next forward pass, making the effective computational depth K × N, where K is the number of reasoning steps and N is the number of layers per step.

In essence, CoT turns a fixed-depth computational circuit into a computational circuit whose depth can grow dynamically. The 2023 paper “Chain-of-Thought Empowers Transformers to Solve Inherently Serial Problems” provides a rigorous mathematical proof of this point.

2.3 Understanding It Through the Attention Mechanism

Every intermediate CoT output becomes a Key and Value for later steps. When the model needs to refer to an earlier calculation during the final step, its Query can attend directly to the exact token position containing that value. This is much more precise than “recalling” it from an ambiguous, distributed representation in hidden layers.

3. CoT’s Threefold Value

4. Ways to Implement CoT

4.1 Zero-Shot CoT

Provide no example; use only one trigger phrase:

"Let's think step by step."              ← The classic form, shown by research to be the most effective
"Please think through this problem one step at a time."
"Think through this carefully."
"Before answering, reason through the problem."

Practical tip: Even when the main prompt is in Chinese, an English CoT trigger often works better. English logical-reasoning text appears far more frequently than Chinese in LLM training data, so the model responds more strongly to “step by step.”

Suitable for: Tasks that vary widely, making it difficult to prepare examples covering every case.

4.2 Few-Shot CoT

Include several examples with their reasoning process in the prompt and ask the model to imitate them:

Question: A pool has two inlet pipes. Pipe A supplies 3 tons per hour, and Pipe B supplies 5 tons per hour.
If both are opened together, how many hours will it take to fill a 40-ton pool?

Reasoning:
- Rate of Pipe A: 3 tons/hour
- Rate of Pipe B: 5 tons/hour
- Combined rate: 3 + 5 = 8 tons/hour
- Required time: 40 ÷ 8 = 5 hours
Answer: 5 hours

Question: {new_question}
Reasoning:

Few-Shot CoT is more reliable than Zero-Shot CoT because you not only tell the model to reason but also demonstrate the format and granularity of the reasoning.

Suitable for: Situations where you can provide high-quality examples that are clearly better than the model’s default behavior.

4.3 Structured CoT (Production-Oriented CoT)

Organize the reasoning with XML, JSON, or other structured tags so code can parse it:

Output your analysis using exactly the following structure:

<thinking>
<symptom_extraction>[Extract key symptoms]</symptom_extraction>
<severity_assessment>[Assess severity]</severity_assessment>
<possible_conditions>[List possible conditions]</possible_conditions>
</thinking>

<answer>[Final answer]</answer>

Suitable for: AI application development that needs to process the reasoning and final answer separately.

4.4 Built-In CoT in Thinking Models

Models such as Claude Extended Thinking, OpenAI o1/o3, and Gemini Thinking Mode are optimized specifically for reasoning through reinforcement learning (RL) during training. They automatically perform deep reasoning in a thinking block and do not require a manual CoT trigger.

Key differences from prompt-based CoT:

5. CoT Prompt Engineering Best Practices

Practice 1: Separate Reasoning from the Final Answer

In production, the reasoning process needs to be stored in logs for debugging and auditing, while the final answer is returned to the user. Use structured tags to separate them:

Analyze this problem.

<thinking>
[Write your reasoning process here]
</thinking>

<answer>
[Provide the final answer here]
</answer>

Practice 2: Specify the Dimensions and Direction of Reasoning

Do not merely say “reason about this.” Give the reasoning process a roadmap:

Analyze whether this code has any problems.

Examine each of the following dimensions in order:
1. First, check logical correctness—does the code implement the intended behavior?
2. Then, check edge cases—empty input, extreme values, concurrency, and so on.
3. Next, check performance—are the time and space complexities reasonable?
4. Finally, check security—are there injection, privilege-escalation, or other risks?

For each dimension, give a judgment first (problem/no problem), then explain the reason.

Principle: If you do not specify dimensions, the model may devote extensive space to the first problem it finds and ignore later dimensions after exhausting its token budget. Specifying dimensions ensures that every critical aspect receives attention.

Practice 3: Self-Consistency Verification

For high-risk reasoning tasks, have the model reason independently several times with temperature > 0, then select the majority answer.

The central idea is that each attempt may follow a different path, but multiple paths are more likely to converge on the correct answer.

# 伪代码
answers = [model.generate(question, temperature=0.7) for _ in range(5)]
final_answer = majority_vote(answers)
confidence = count(final_answer) / 5

Suitable for: High-risk settings such as medicine, finance, and law, where the method doubles cost but can improve accuracy by 5–15%.

Practice 4: Give the Model an Opportunity to Reflect

Ask the model to inspect its work after reasoning:

Step 1: Write your reasoning process and preliminary answer.

Step 2: Review your reasoning and check for the following problems:
- Are there any calculation errors?
- Did you overlook a condition?
- Is the conclusion reasonable?

If you find a problem, correct it and provide the final answer.

Principle: The context in the reflection stage already contains the complete reasoning chain. The model can use another forward pass to “inspect” its previous reasoning, much like a person checking a completed mathematics problem.

Practice 5: Adjust the Strategy for Thinking Models

For thinking models such as Claude Extended Thinking and OpenAI o1/o3:

Do not trigger CoT manually with phrases such as “Let’s think step by step.” This may interfere with an already optimized reasoning strategy and reduce performance.

Do describe the task and expected output format clearly:

Analyze all security vulnerabilities in the following code.
For every vulnerability, provide its type, severity (high/medium/low), and a recommended fix.
Output the result as a JSON array.

The model automatically performs deep reasoning in its thinking block. You need only focus on the clarity of the task description.

6. Controlling the Quality of Few-Shot CoT

6.1 A Bad Example Is Worse Than No Example

Few-shot prompting is a double-edged sword. The model uses attention to match and copy patterns. It does not distinguish a good pattern from a bad one; it imitates both faithfully.

Poor examples cause three kinds of harm:

A ceiling on reasoning granularity: If an example shows only a crude two-step process, the model may also use only two steps even when the problem requires five.

Misdirected reasoning: If the example skips steps or contains logical gaps, the model learns those bad habits.

Excessive format constraints: If every example uses one specific format, the model becomes locked into that format even when another would suit the current problem better.

6.2 Decision Criteria

The prerequisite for using Few-Shot CoT is that you can provide examples clearly better than the model’s default behavior, such as expert reasoning in a particular domain or an output format that must be followed strictly.

Zero-Shot CoT is more appropriate when you are uncertain what the optimal reasoning path is or when the tasks vary widely. Omitting examples gives the model more freedom in these cases.

Core principle: if you are not sure what the reasoning process for the “reference answer” should look like, do not provide an example; let the model work it out.

6.3 How to Ensure Quality When Few-Shot Is Required

Recommended method: “Have the model help generate and filter examples.”

1. Ask the model to perform Zero-Shot CoT on a batch of questions.
2. Have a person verify which reasoning processes are correct and high-quality.
3. Use the verified examples as Few-Shot examples.

This is more effective than writing every example from scratch because the model’s generated reasoning style aligns with its own “thinking habits.”

7. CoT’s Limitations and Costs

8. A Decision Tree for Using CoT

Are you using a Thinking model (o1/o3, Claude Extended Thinking, Gemini Thinking)?
│
├── Yes → Do not trigger CoT manually
│         Focus on a clear task description and output format
│
└── No → Does the task require multi-step reasoning?
    │
    ├── No → Do not use CoT; answer directly
    │        (fact retrieval, translation, creative writing, and so on)
    │
    └── Yes → Do you have high-quality reasoning examples?
        │
        ├── Yes → Use Few-Shot CoT
        │         + specify reasoning dimensions
        │         + separate thinking and answer
        │
        └── No → Use Zero-Shot CoT
                 ("Let's think step by step")
                 │
                 └── Is this a high-risk setting?
                     │
                     ├── Yes → Add Self-Consistency
                     │         + reflection checks
                     │
                     └── No → Basic CoT is sufficient

9. Relationship with Other Prompt Engineering Techniques

CoT is not an isolated technique. It works together with other core prompt engineering techniques:

Clear Instructions + CoT: Specifying reasoning dimensions applies Clear Instructions to a CoT scenario.

Few-Shot + CoT: Few-Shot CoT is the natural combination of Few-Shot and CoT. The examples demonstrate not only input and output but also the reasoning process.

CoT + delimiters: Separating reasoning and answers with XML tags applies delimiter techniques to a CoT scenario.

Relationship between CoT and Context Engineering: Reasoning tokens generated by CoT occupy space in the context window. When an Agent runs over a long period, historical reasoning needs to be compressed or summarized to prevent context-window overflow. This is one of Context Engineering’s central challenges.

10. Key Papers

1. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., 2022) — The foundational CoT paper
2. Large Language Models are Zero-Shot Reasoners (Kojima et al., 2022) — Discovered the effectiveness of “Let’s think step by step”
3. Self-Consistency Improves Chain of Thought Reasoning (Wang et al., 2023) — The method of sampling multiple times and choosing the majority answer
4. Chain-of-Thought Empowers Transformers to Solve Inherently Serial Problems (Feng et al., 2023) — Theoretical proof that CoT breaks through fixed-depth limitations