iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Chain-of-thought (CoT) prompting asks a large language model to produce intermediate steps before its answer, often by showing it worked examples. Research found substantial gains on some reasoning benchmarks, but the results depend on the model, task, and prompt—and a convincing written explanation is not proof that the answer is correct.
What is chain-of-thought prompting?
In ordinary prompting, a model may receive a question and be asked for a final answer. With CoT prompting, the prompt encourages it to include intermediate reasoning steps on the way to that answer. The aim is to make multi-step tasks easier for the model to handle by eliciting a sequence of intermediate outputs rather than asking for the result in one jump.
The original method used few-shot examples: the prompt included sample questions paired with answers that showed intermediate steps. It changed the prompt, not the model’s weights. Google Research researchers Jason Wei and Denny Zhou described the approach this way: “such thought processes can be elicited by including a few examples of chain of thought via prompting only,” without modifying the model’s weights. See the NeurIPS 2022 paper and Google Research’s overview.
Recommended Free Tools
How do the main approaches differ?
| Approach | What the prompt or method does | Published result in the cited 2022 work |
|---|---|---|
| Few-shot CoT | Includes worked examples whose answers show intermediate steps. Ordinary few-shot prompting instead gives examples with final answers but no reasoning demonstrations. | Google Research reported 58% accuracy on GSM8K for PaLM with 540 billion parameters and eight CoT exemplars. The overview says the comparison used an external calculator for basic arithmetic functions. |
| Zero-shot CoT | Adds an instruction such as “Let’s think step by step,” without hand-crafted CoT examples. | Kojima et al. reported that InstructGPT (text-davinci-002) accuracy rose from 17.7% to 78.7% on MultiArith and from 10.4% to 40.7% on GSM8K in their specific evaluation. |
| Self-consistency | Samples multiple reasoning paths instead of relying on one greedy path, then selects the answer that is most consistent across those samples. | Google Research’s overview gives a rounded 74% GSM8K accuracy for the self-consistency follow-up. Its paper summary reports gains of GSM8K +17.9%, SVAMP +11.0%, AQuA +12.2%, StrategyQA +6.4%, and ARC-challenge +3.9%. |
These are results from separate 2022 studies and evaluations, not a head-to-head comparison under one shared protocol. The 74% figure is the overview’s rounded result; the benchmark-specific deltas are reported separately in the self-consistency paper summary. They should not be read as current performance estimates for arbitrary models or prompts.
#1 Best Overall
What did the studies establish—and what did they not?
CoT can help on particular reasoning benchmarks
The foundational work evaluated arithmetic, commonsense, and symbolic reasoning. Its results were strongest with sufficiently large models; the study does not establish that CoT improves every model, task, or prompt. The zero-shot study likewise reported gains on some tasks rather than a uniform improvement. Its InstructGPT figures above belong to the named model and evaluation, not to LLMs generally. The original NeurIPS paper and Kojima et al.’s zero-shot-CoT paper describe those experimental settings.
A generated trace is not independent verification
A model’s written reasoning is generated text. The cited CoT studies measure task performance; they do not establish that every displayed trace faithfully records the internal process that produced an answer. A plausible explanation can accompany a wrong result, so check consequential answers against reliable evidence or an independent calculation rather than treating the trace itself as proof. Anthropic’s discussion of imperfectly calibrated model self-evaluation concerns a distinct issue; it is not a direct test of CoT faithfulness: Language models (mostly) know what they know.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.When should you use chain-of-thought prompting?
CoT is most relevant when a task plausibly requires several steps and you want the model to expose intermediate work for inspection. For few-shot CoT, the examples should resemble the task and demonstrate the kind of intermediate steps you want. Zero-shot CoT is a simpler prompt variation, but the study results do not guarantee an improvement on a new task. Self-consistency generates several paths and selects a recurring answer, so it requires multiple generations rather than a single one.
When judging whether an approach helped, compare it with a baseline on the same task and model. Record whether examples were supplied, how many reasoning paths were generated, how the final answer was selected, and the evaluation measure. The cited abstracts and summaries do not provide a common current comparison of cost or latency, so no general efficiency advantage can be inferred from these accuracy results.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
