Chain-of-Thought Prompting

Chain-of-thought (CoT) prompting makes a language model write intermediate steps before its final answer, instead of answering directly. Wei et al. introduced it by adding worked examples that show the reasoning, not only the answer, to few-shot prompts. Kojima et al. showed that appending a phrase such as “Let’s think step by step” can trigger similar behavior with no examples. Today the behavior is also trained into models, so it is often produced without prompting.

Why it can help

In ordinary autoregressive decoding, a transformer runs a forward pass for each new token. Producing intermediate steps adds passes; their cost also depends on context length, since attention reads the available keys and values. A question such as “what is 47 times 6 plus 13?” asked for a one-token answer must be resolved in that single pass. If the model first writes

47 x 6 = 282
282 + 13 = 295

then each line is a smaller lookup, and the written result of the first line is in the context for the second. The generated text works as external working memory, and it spends more tokens, hence more computation, on the problem. The benefit depends on the problem being decomposable into steps that are each easier than the whole.

What the evidence says

CoT is not a general accuracy boost. Wei et al. report large gains for large models on arithmetic, commonsense, and symbolic reasoning benchmarks, and also report that the effect appears only at sufficient model scale. Sprague et al., in a meta-analysis of many papers and their own experiments, find strong benefit mainly on tasks involving math or logic and small gains elsewhere, and on one knowledge benchmark nearly identical accuracy with and without CoT unless the question involved an equals sign. A reasonable default is therefore to test CoT on the task, expecting it to matter for multi-step calculation and symbolic work and to matter little for recall or classification.

CoT can also hurt. Each written step is another sample from the model, which can be wrong. With a hypothetical 98% chance of each step being correct, a 3-step chain is fully correct with probability \(0.98^3 \approx 0.94\) and a 5-step chain with probability \(0.98^5 \approx 0.90\). If the direct answer would have been right more often than that, the chain loses. The error also propagates, because every later line is conditioned on the earlier text.

The text is not a faithful trace

A generated rationale reads like an explanation, and it is tempting to treat it as the model’s actual reason for the answer. The evidence does not support that reading. Turpin et al. add biasing features to prompts, such as always placing the correct option in the same position in the few-shot examples, and find that models change their answers while the written reasoning does not mention the bias. Lanham et al. test faithfulness by editing or truncating the reasoning and checking whether the answer changes, and find that how much the answer depends on the reasoning varies by task and model. Practical consequences: use the chain as a thing to check (verify each step against a source or a program), not as proof of how the answer was reached, and do not present it to users as an audit trail.

Costs and variations

The chain consumes output tokens, and output tokens cost latency and money. A hypothetical answer of 5 tokens that becomes a 125-token rationale plus answer is a 25-fold increase in generated tokens. Options that trade this off:

• Self-consistency samples several chains and takes the most common final answer, which multiplies cost by the number of samples and helps when errors are not all the same error.
• Hidden or summarized reasoning keeps the chain out of the user-facing output while still spending the tokens.
• Selective use applies CoT only to question types where it was shown to help.

References

• Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — https://arxiv.org/abs/2201.11903
• Sprague et al., To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning — https://arxiv.org/abs/2409.12183
• Turpin et al., Language Models Don’t Always Say What They Think — https://arxiv.org/abs/2305.04388