Chain of Thought
Chain-of-thought prompting asks a language model to produce intermediate reasoning steps before its final answer. Wei et al. (2022) introduced the technique and showed that exemplars carrying rationales, not only input-output pairs, improved arithmetic, commonsense, and symbolic reasoning on sufficiently large models. The gain was an emergent property of scale. Smaller models gained little or nothing from the same prompting style.
The one idea that generates the whole space
Every variant of chain of thought starts from a single move: make the model show its work. Exposing the intermediate steps gives the model room to reach an answer it would miss in one jump, and it gives you a surface to inspect.
That move alone has a weakness. The work the model shows is unverified text. A wrong step propagates silently, and the rationale itself reads as confident whether or not it tracks the true computation. Every successor to plain chain of thought is an answer to the same question: now that the model is showing its work, how do you make that work trustworthy? Four answers generate the field.
The rest of this page works outward from that generator. First the two base recipes, then where the move helps and where it hurts, then the faithfulness caveat that motivates the four structures, then the structures themselves.
Two base recipes: few-shot and zero-shot
Two recipes carry the name "chain of thought." They behave differently and warrant separate treatment.
Few-shot CoT is the original recipe from Wei et al. (2022). The prompt includes several worked examples, each a problem, a step-by-step rationale, and an answer. The model imitates that structure on a new problem. This form depends on the exemplars. The rationales need to be correct, consistent in format, and representative of the problems the system sees in production. The original paper documents that exemplar quality and consistency materially affect the gains.
Zero-shot CoT comes from Kojima et al. (2022), who showed that a single instruction ("Let's think step by step.") appended to the question elicits multi-step reasoning on many tasks without hand-crafted exemplars. Zero-shot CoT costs less to maintain and applies across diverse tasks. The trade-off is variance. The reasoning structure is less constrained, and the outputs are harder to parse downstream. Modern instruction-tuned models often produce step-by-step reasoning when the task warrants it with no explicit cue. The cue still helps on borderline cases and on tasks where the model's default is to answer directly.
| Recipe | Source | Prompt content | Strength | Cost |
|---|---|---|---|---|
| Few-shot CoT | Wei et al. (2022) | Worked examples with rationales | Controlled, consistent reasoning structure | Exemplar curation. Format depends on rationale quality |
| Zero-shot CoT | Kojima et al. (2022) | A single cue, no exemplars | Cheap to maintain, broad coverage | Higher variance, harder to parse |
When the move helps
The original paper and later benchmarks point to chain of thought helping when the task:
- Requires multiple inference steps to reach the answer.
- Has intermediate quantities, sub-results, or named entities the model needs to track.
- Benefits from the model exposing its working, so an error in one step does not silently corrupt the final answer.
- Runs on a model capable enough to hold coherent reasoning across the steps. The emergent-scale finding from Wei et al. (2022) still applies. The smallest models reason worse with CoT than without.
Math word problems, multi-hop question answering, planning tasks, and symbolic manipulation are the canonical fits.
When the move does not help, or hurts
Chain of thought is not universally beneficial. Four patterns are worth carrying.
- Single-step or pattern-matching tasks. Classification, simple extraction, and short transformations gain little. The extra tokens cost latency and money without improving accuracy.
- Tasks the model solves without working. A model that already knows the answer to a trivia question does not benefit from deriving it. The derivation often introduces errors the direct answer did not have.
- Tasks where the failure mode is knowledge, not reasoning. CoT does not manufacture facts the model lacks. A reasoning scaffold on a knowledge-gap problem produces fluent-but-wrong rationales. The fix is retrieval, not reasoning.
- Tasks with strict latency or token budgets. CoT inflates output length by an order-of-magnitude factor compared to a direct answer, because the model emits the reasoning chain before the conclusion. On a 30-second user-facing budget, the cost often outweighs the accuracy gain.
The faithfulness caveat
The move's central caveat comes from Turpin et al. (2023), titled Language Models Don't Always Say What They Think. The paper shows that chain-of-thought rationales do not always reflect the true cause of a model's prediction. The model produces a fluent rationale while the actual decision pathway runs elsewhere.
This does not make CoT bad. It makes CoT a performance technique with a misleading side-effect: outputs that look more inspectable than they are. The unfaithfulness is the reason the four structures below exist. Each one replaces "trust the visible chain" with a check the chain cannot fake.
The four structures: verify, decompose, search, execute
Each successor takes the base move and adds one of the four structures from the generator. The table is the reference spine. The prose after it connects the four to the failure they each answer.
| Structure | Pattern | Source | What it does | Cost |
|---|---|---|---|---|
| Verify | Self-consistency | Wang et al. 2022 | Sample multiple reasoning chains, return the majority answer | N× more LLM calls |
| Decompose | Least-to-most prompting | Zhou et al. 2022 | Break a hard problem into easier sub-problems, then solve in order | Extra prompting stage. Improves easy-to-hard generalization |
| Search | Tree of Thoughts | Yao et al. 2023 | Search over candidate intermediate states with self-evaluation and backtracking | Many more calls. Suits planning and puzzles |
| Execute | Program-aided language models | Gao et al. 2022 | Generate a runnable program as the reasoning trace, delegate computation to an interpreter | Needs an execution environment. Sidesteps arithmetic errors entirely |
Self-consistency answers unfaithfulness by majority vote. One chain rationalizes. Agreement across many chains is harder to fake. Least-to-most answers it by shortening each step, so a wrong sub-result surfaces against an easier check. Tree of Thoughts answers it by making the search explicit, so a dead-end branch gets evaluated and abandoned instead of carried forward. Program-aided language models answer it by moving the computation out of the text entirely, so the interpreter, not the rationale, produces the number.
Putting the model to work
A few rules of thumb follow from the generator.
- Try the task without CoT first on a recent instruction-tuned model. Modern models already reason when the task warrants it.
- If CoT helps, decide whether the rationale needs to be visible to the user. The right architecture is often "reason internally, return only the answer" through a system-prompt instruction or the model's reasoning-tokens feature where supported.
- For exact arithmetic, symbolic logic, or executable answers, prefer program-aided language models or tool use over pure CoT. The reasoning trace then has a checkable artifact.
- Treat the rationale as a debugging aid, not as proof. Combine it with evaluation against ground truth.
Related
- Prompt Engineering. The broader cluster CoT sits inside.
- Few-Shot Prompting. The exemplar mechanism CoT builds on.
- Prompt Evaluation. How to tell whether CoT helps your task.