When Chain-of-Thought Hurts

Chain-of-thought (CoT) prompting asks a model to write intermediate steps before its final answer. It became a prominent technique after research reported benefits on selected multi-step reasoning benchmarks. But “ask for reasoning” is not a universal improvement. A 2025 ICML paper evaluated six tasks drawn from psychological studies where deliberation can hurt human performance. The researchers found significant CoT-related drops for state-of-the-art models on three tasks, while results on the other tasks were mixed. That study gives evidence that performance can fall in particular settings; it does not show that CoT generally harms models or identify one rule that predicts every task. Extra written steps also have a practical cost: they use output space and may increase latency. More text can introduce an unsupported assumption that the model then carries into its answer. An explanation should not be mistaken for a faithful record of the internal process or proof that a conclusion is correct. For reasoning models, the appropriate prompting advice can differ. OpenAI’s current API guide, for example, recommends avoiding “think step by step” instructions for its reasoning models, while its model-specific guidance for some non-reasoning models may discuss other prompting approaches. Follow the documentation for the model being tested. Choose based on evidence from the task. Compare direct and CoT variants on the same representative examples, use a predefined scoring rule, and include latency or token limits if they matter to the application. Keep an approach only when it improves the required outcomes without unacceptable costs. For high-stakes decisions, independent checks and expert review matter more than whether the model displays an explanation. Avoid assuming that a longer rationale is inherently more transparent, reliable, or safe.