数学的推論のプロセス監視

When a language model solves a math problem, it usually writes a chain of steps before giving an answer. There are two ways to give feedback on such a solution. Outcome supervision checks only the final answer: right or wrong. Process supervision checks each intermediate step: is this line correct, and does it move toward a solution? The feedback is used to train a reward model, a separate network that scores solutions. A reward model trained on step labels is called a process reward model (PRM); one trained on final-answer labels is an outcome reward model (ORM). Reward models can then be used to pick the best of several candidate answers, a verifier role, or to provide training signals for reinforcement learning.

The motivation is easy to see with an example. A model can reach the correct number through a wrong argument, such as two errors that cancel, or a lucky guess. Outcome supervision rewards that solution anyway. It also gives only one bit of information for a long solution, so the learner cannot tell which step went wrong, a problem known as credit assignment. Process supervision points to the exact step that failed.

An early large comparison came from Jonathan Uesato, Geoffrey Irving, Irina Higgins and colleagues in a paper posted November 25, 2022, using GSM8K, a set of grade-school word problems. They found that outcome-based supervision gave similar final-answer error rates with fewer labels. However, getting the reasoning itself right needed process-based feedback or reward models that imitate it. Their best setup cut final-answer error from 16.8% to 12.7%, and cut reasoning errors among solutions with correct final answers from 14.0% to 3.4%.

OpenAI's 'Let's Verify Step by Step', by Hunter Lightman and colleagues and posted May 31, 2023, scaled the question up. The team fine-tuned models from base GPT-4, trained a generator to write solutions one step per line, and had human labelers rate each step of solutions to problems from MATH, a benchmark of competition mathematics. A step was positive if correct and reasonable, negative if incorrect or unreasonable, and neutral if ambiguous. Labeling stopped at the first mistake, so for a wrong solution the process label adds one fact the outcome label lacks: where the error is. The resulting dataset, PRM800K, contains about 800,000 step labels across 75,000 solutions to 12,000 problems.

The headline result comes from best-of-N search. For each of 500 held-out MATH test problems, the generator produced up to 1,860 solutions; each reward model picked its top-ranked solution, which was graded on its final answer. The PRM chose a correct solution 78.2% of the time, the ORM 72.4%, and majority voting, which picks the most common final answer, 69.6%. The gap grew as more candidates were sampled. On 224 newer AP Calculus, Chemistry, Physics and AMC questions released after the pretraining data was gathered, the PRM scored 72.9% in aggregate versus 63.8% for the ORM and 61.3% for majority voting. The authors also reported that active learning, showing labelers convincing wrong-answer solutions that fooled the current PRM, was about 2.6 times more data-efficient than labeling solutions at random.

Human labels are expensive, so later work tried to generate step labels automatically. Math-Shepherd, posted December 14, 2023 by Peiyi Wang and colleagues, labels a step by letting a model, called a completer, finish the solution from that step several times. A step that often leads to the right answer is treated as good. Trained this way without human annotation, Math-Shepherd raised Mistral-7B from 28.6% to 33.0% on MATH through step-level reinforcement learning, and to 43.5% when also used to verify candidate answers.

Process supervision has clear limitations. The DeepSeek-R1 paper, posted January 22, 2025, lists PRMs among its unsuccessful attempts and names three problems: it is hard to define what counts as a step in general reasoning, hard to judge whether an intermediate step is correct, and a learned reward model invites reward hacking, where the policy learns to please the verifier rather than be right. The authors concluded that PRMs work well for reranking and guided search but did not justify their added compute in large-scale reinforcement learning, and they trained with rule-based answer checks instead. Lightman and colleagues also note open issues: they could not rule out some test-set contamination, they studied only math, and their PRM is itself a model that can be wrong. Automatic labels like Math-Shepherd's contain noise, and the paper lists the cost of the completion process as a limitation.

In practice, process supervision needs three things: problems with checkable answers, solutions split into clear steps, and a labeling budget, human or computational. Using a PRM at inference time also costs compute, since best-of-N requires generating and scoring many candidates. Lightman and colleagues argued the method carries a negative alignment tax: it improved accuracy while rewarding reasoning that humans endorse, which they view as a safety benefit. Whether that holds outside math is, in their words, unknown.