📋 KEY INSIGHTS
- LLM reasoning refers to the capacity of language models to solve problems that require multi-step logical inference, mathematical computation, planning, or code execution — going beyond pattern-matching over training data to perform structured deduction. The field has undergone a transformation since late 2024 with the emergence of “thinking” models (OpenAI o1/o3, DeepSeek-R1, Gemini 2.0 Flash Thinking, Claude 3.5 Sonnet extended thinking) that explicitly allocate additional test-time computation to reasoning before producing a final answer. These models represent a shift from the traditional paradigm (scale training compute to increase capability) to a new paradigm (scale inference compute to increase reasoning quality), with implications for how we train, evaluate, and deploy LLMs.
- Chain-of-Thought (CoT) prompting (Wei et al., NeurIPS 2022) demonstrated that adding “Let’s think step by step” or providing few-shot examples of intermediate reasoning steps dramatically improves LLM performance on multi-step reasoning benchmarks. The key finding: CoT is an emergent capability that only appears at sufficient model scale (~100B parameters for few-shot CoT, ~8B for zero-shot CoT with the magic phrase). The mechanistic explanation is debated — CoT may work by: (1) decomposing a hard multi-step problem into easier single-step subproblems each within the model’s capability; (2) providing a “scratchpad” that externalises intermediate computations in the context window; (3) activating reasoning-associated activation patterns that are normally suppressed by the next-token prediction objective on answer tokens. Self-consistency (Wang et al., 2022) improves CoT further by sampling k CoT trajectories and taking the majority vote on the final answer, achieving consistent accuracy improvements on GSM8K, MATH, and commonsense reasoning benchmarks.
- Tree-of-Thought (ToT, Yao et al., NeurIPS 2023) extends CoT from linear reasoning chains to tree-structured search: at each reasoning step, generate B candidate next thoughts (a branching factor of B), evaluate each thought’s promise (using the LLM as a heuristic evaluator), and select the most promising branches for continued expansion. ToT enables backtracking when a reasoning path reaches a dead end — unlike CoT which is committed to the linear chain once generated. ToT with BFS or DFS search over the reasoning tree achieves state-of-the-art on tasks requiring global planning (Game of 24, creative writing with constraints) where linear reasoning fails. The connection to our RL guide is direct: ToT’s tree search is a form of Monte Carlo Tree Search (MCTS) over the reasoning space, where the value function is the LLM’s self-evaluation of partial reasoning chains — the same MCTS framework used for game-playing agents in AlphaGo and AlphaCode.
- Process Reward Models (PRMs, Lightman et al., 2023) train a reward model that evaluates the quality of intermediate reasoning steps (not just final answers), enabling more precise supervision of the reasoning process. PRM800K is the dataset of 800K step-level human annotations (correct / incorrect / neutral) for mathematical reasoning traces on MATH problems, used to train process reward models for math. PRMs enable two applications: (1) best-of-N selection — generate N reasoning traces, score each step of each trace with the PRM, select the trace with the highest product of step-level scores; (2) RLHF training signal — use step-level PRM scores as the reward signal for PPO or GRPO training, providing denser and more informative reward than outcome-only reward models. The connection to our reward modelling guide is direct: PRMs are the step-level extension of the outcome reward models described there — both are preference-trained discriminators that score LLM outputs, but PRMs assign scores to each step rather than the full output, enabling fine-grained credit assignment in reasoning chains.
- DeepSeek-R1 (DeepSeek, January 2025) demonstrated that strong mathematical and coding reasoning can be achieved through pure RL training without supervised fine-tuning on reasoning traces — a key methodological breakthrough. The DeepSeek-R1-Zero model was trained with GRPO (Group Relative Policy Optimisation, a variant of PPO with no critic network) using only outcome reward (correct/incorrect final answer) on mathematical and coding problems. Despite no explicit supervision of intermediate steps, R1-Zero spontaneously developed extended reasoning chains with self-reflection, self-correction, and verification steps — an emergent reasoning behaviour arising from RL optimisation of the outcome reward. DeepSeek-R1 (the full model) adds a cold-start phase with a small set of high-quality supervised reasoning traces to stabilise early training, then continues with GRPO. R1 achieves performance competitive with o1 on AIME 2024, Codeforces, and MATH benchmarks at a fraction of the training compute.
- Test-time compute scaling is the paradigm shift: for a fixed model size and training budget, reasoning performance can be substantially improved by allocating more inference compute — through more CoT tokens, more tree search branches, or more RL-guided rollouts. The Scaling Laws for Test-Time Compute (Snell et al., 2024) paper shows that scaling test-time compute is more efficient than scaling model parameters for hard reasoning problems: a small model with 10× test-time compute often outperforms a 10× larger model with 1× compute. This has significant infrastructure implications for LLM deployment — reasoning-heavy workloads (math tutoring, code generation, formal verification) benefit from dynamic compute allocation based on problem difficulty, using the same compute budget for a simple query as a standard model but allocating 100× more tokens and tree-search rollouts for hard problems. The connection to our LLM evaluation guide is direct: evaluating thinking models requires tracking not just final answer accuracy but reasoning trace quality (step correctness, logical consistency, search efficiency) — the extended benchmarks AIME 2024, LiveCodeBench, and GPQA Diamond measure exactly the hard reasoning tasks where test-time compute scaling provides the largest gains.
The history of AI reasoning is a history of ever-more-powerful search and inference: from symbolic theorem provers and expert systems in the 1970s, to Monte Carlo Tree Search enabling superhuman Go play in the 2010s, to large language models with chain-of-thought prompting in 2022, to dedicated “thinking” models with explicit test-time compute allocation in 2024-2025. The fundamental insight connecting all these threads is that hard problems require structured exploration of a solution space — a single forward pass through any model, however large, is insufficient for problems requiring backtracking, verification, or multi-step deduction. This guide covers the full LLM reasoning landscape: chain-of-thought prompting and self-consistency, Tree-of-Thought search, Process Reward Models for step-level supervision, GRPO and RL-based reasoning training (DeepSeek-R1), test-time compute scaling, and the emerging ecosystem of reasoning benchmarks and evaluation frameworks.
Chain-of-Thought, Tree-of-Thought and Process Reward Models
Chain-of-Thought prompting works by eliciting the model to verbalise its reasoning process before producing the final answer, decomposing a multi-step problem into a sequence of natural-language inference steps. For mathematical reasoning, CoT converts “What is 15% of 240?” from a direct recall problem (what is 15×240/100?) into a step-by-step computation: “15% = 15/100 = 0.15; 0.15 × 240 = 36”. For multi-hop factual reasoning, CoT provides intermediate conclusions that serve as context for subsequent deductions. The emergence threshold (~100B parameters for few-shot CoT) reflects a minimum model capability requirement: the model must be able to reliably execute each individual reasoning step for CoT to help; if individual steps fail, chaining them amplifies errors rather than reducing them. The connection to our prompt engineering guide is direct: CoT is the most impactful prompting technique described there, and the variants (zero-shot CoT, few-shot CoT, self-consistency, structured CoT) represent the core prompting toolkit for reasoning-heavy tasks. Self-consistency exploits the probabilistic nature of LLM generation: by sampling k diverse CoT trajectories (using temperature > 0 to introduce variation), different reasoning paths may reach the same correct answer through different routes, and the majority vote aggregates the independent evidence from each path. Self-consistency consistently improves accuracy by 5-10 absolute percentage points on GSM8K, MATH, and commonsense benchmarks across model sizes. The limitation: sampling k trajectories costs k× the inference compute, and majority voting does not propagate reasoning to verify which trajectories are correct — it only selects answers that are frequently reached. Tree-of-Thought addresses this limitation by explicitly searching the reasoning tree rather than generating independent chains: at each step, propose B thoughts (candidate next reasoning steps), evaluate each thought using the LLM’s self-evaluation capability (prompt the model: “Given this partial reasoning, is this step correct and promising? Rate 1-5”), and select the top-k thoughts for further expansion. ToT with BFS uses a fixed compute budget to explore the shallowest promising nodes first (useful for well-structured problems); ToT with DFS commits to the most promising branch and backtracks only on failure (useful for problems with early hard constraints). The connection to our Monte Carlo methods guide is direct: ToT with random rollout value estimation is exactly MCTS applied to the reasoning space — the four phases of MCTS (selection, expansion, simulation, backpropagation) map directly to ToT’s thought selection, thought generation, rollout to completion, and score backpropagation. PRMs provide the value function that makes this MCTS search tractable: rather than simulating each partial reasoning chain to completion (expensive), the PRM estimates the probability of reaching a correct final answer from the current step, enabling efficient branch evaluation without full rollout. The PRM800K training procedure: collect LLM-generated reasoning traces on MATH problems, have human annotators label each step as correct/incorrect/neutral, and train a step-level reward model to predict step correctness. Best-of-N with PRM achieves substantially higher accuracy than best-of-N with outcome reward model (ORM) on MATH, since the PRM can identify and exclude traces that contain early reasoning errors even when they happen to arrive at the correct final answer by coincidence.
| Method | Compute Type | Backtracking? | Value Signal | Best For |
|---|---|---|---|---|
| Chain-of-Thought | Single pass + tokens | No | None | Medium difficulty; emergent |
| Self-Consistency | k parallel passes | No | Majority vote | Well-defined answers |
| Tree-of-Thought (BFS/DFS) | Tree search | Yes | LLM self-eval / PRM | Planning; structured problems |
| MCTS + PRM | Rollout + search | Yes | Process Reward Model | Math; competitive coding |
| o1/R1 extended thinking | Variable <think> tokens | Implicit | GRPO/PPO outcome reward | AIME; Codeforces; GPQA |
DeepSeek-R1, GRPO Training and Test-Time Compute Scaling
DeepSeek-R1-Zero’s training methodology is conceptually clean: start from a base language model (DeepSeek-V3), define a reward function that returns +1 for correct final answers and 0 for incorrect ones (with format penalties for violating the response structure), and apply GRPO to maximise expected reward over a distribution of mathematical and coding problems. GRPO (Group Relative Policy Optimisation) generates G rollouts per prompt, computes each rollout’s reward, normalises rewards within the group (subtracting group mean, dividing by group std), and uses these normalised rewards as advantage estimates for the policy gradient update — eliminating the need for a separate critic/value network compared to PPO. The emergence of self-reflection in R1-Zero (the model spontaneously generating phrases like “Wait, let me reconsider…” and “Let me verify this step…”) is striking evidence that meta-cognitive reasoning behaviours can emerge from outcome-only RL training without explicit supervision of these behaviours. The connection to our RLHF guide is direct: GRPO is a variant of the PPO-based RLHF training described there — both fine-tune LLMs with policy gradient methods to maximise reward model scores, but GRPO is specifically designed for multi-rollout reasoning training and avoids the critic variance that makes PPO expensive at LLM scale. Test-time compute scaling laws reveal a key inflection point: for problems at the frontier of model capability (problems the model gets right 20-50% of the time with greedy decoding), test-time compute scaling (more sampling, more search) provides log-linear accuracy improvements. For problems well within model capability (>80% greedy accuracy) or well outside it (<5%), test-time compute scaling provides diminishing returns. This suggests that compute-optimal inference allocates most test-time compute to problems in the "difficult but solvable" regime — a difficulty-aware routing system (use standard decoding for easy queries, thinking mode for hard queries) can achieve the same average accuracy as always using thinking mode at a fraction of the average cost. The connection to our LLM evaluation guide is direct: evaluating thinking models requires new benchmarks that are genuinely hard for frontier models — AIME 2024 (competition mathematics, designed to have only a few correct answers even among top human students), GPQA Diamond (graduate-level physics, chemistry, and biology questions), and LiveCodeBench (competitive programming from contests after the model’s training cutoff) are the standard benchmarks for measuring reasoning model capability, and the LLM evaluation framework described there covers the contamination-aware evaluation methodology required for fair comparison. The connection to our AI agents guide is direct: reasoning models are increasingly used as the planning component of agentic systems — the same test-time compute scaling that improves mathematical reasoning also improves multi-step tool use and planning in agent frameworks, where the LLM must reason about which tool to call next, what arguments to provide, and how to interpret tool outputs in service of a long-horizon goal.
✦ SUMMARIZE THIS ARTICLE WITH AI
Prompt engineering (chain-of-thought as a prompting technique; zero-shot and few-shot CoT) is in our Prompt Engineering guide. RLHF (PPO/GRPO for LLM fine-tuning; reward models) is in our RLHF guide. Reward modelling (PRMs as step-level reward models; outcome vs. process reward) is in our Reward Modelling guide. LLM evaluation (AIME, LiveCodeBench, GPQA Diamond benchmarks for thinking models) is in our LLM Evaluation guide. AI agents (reasoning models as planners in agentic systems) are in our AI Agents guide. RL for real-world applications (MCTS for ToT; policy gradient for R1 training) is in our RL guide. Monte Carlo methods (MCTS applied to reasoning tree search) are in our Monte Carlo guide.



