āĻ­āĻžāώāĻž āĻāφāχ āĻ—āĻžāχāĻĄ

āĻ—āĻŖāĻŋāϤ āϝ⧁āĻ•ā§āϤāĻŋāϰ āϜāĻ¨ā§āϝ āĻĒā§āϰāĻ•ā§āϰāĻŋāϝāĻŧāĻž āϤāĻ¤ā§āĻ¤ā§āĻŦāĻžāĻŦāϧāĻžāύ

Process supervision trains a reward model to judge each step of a worked solution, instead of only checking the final answer.

āĻāχ āύāĻŋāĻ°ā§āĻĻ⧇āĻļāĻŋāĻ•āĻž āĻļ⧁āύ⧁āύāĻāφāχ-āĻŦāĻ°ā§āĻŖāĻŋāϤ ¡ ~7 min

āĻāχ āĻĒ⧃āĻˇā§āĻ āĻžāϝāĻŧā§Ž āĻŽāĻŋāύāĻŋāϟ āĻĒāĻĄāĻŧāĻž
  1. āĻ“āĻ­āĻžāϰāĻ­āĻŋāω
  2. āĻŽā§‚āϞ āĻŸā§‡āĻ•āĻ“āϝāĻŧ⧇
  3. āĻ—āĻ­ā§€āϰ āĻĄā§āĻŦ
  4. Scoring two solutions with step-level probabilities
  5. āĻŦāĻžāĻ¸ā§āϤāĻŦ-āĻŦāĻŋāĻļā§āĻŦ āĻŦāĻžāĻ¸ā§āϤāĻŦāĻžāϝāĻŧāύ
  6. āĻ—āĻŖāĻŋāϤ āϝ⧁āĻ•ā§āϤāĻŋāϰ āϜāĻ¨ā§āϝ āĻĒā§āϰāĻ•ā§āϰāĻŋāϝāĻŧāĻž āϤāĻ¤ā§āĻ¤ā§āĻŦāĻžāĻŦāϧāĻžāύ⧇āϰ āĻ­āĻŦāĻŋāĻˇā§āϝāϤ
  7. āĻāϟāĻžāϕ⧇ āĻŦāĻžāĻ¸ā§āϤāĻŦ⧇ āĻĒā§āϰāϝāĻŧā§‹āĻ— āĻ•āϰāĻž āĻšāĻšā§āϛ⧇
  8. āϏ⧂āĻ¤ā§āϰ āĻāĻŦāĻ‚ āφāϰāĻ“ āĻĒāĻĄāĻŧ⧁āύ
  9. āĻ…āĻ¨ā§āĻŦ⧇āώāĻŖ āϚāĻžāϞāĻŋāϝāĻŧ⧇ āϝāĻžāύ
  10. āĻĒā§āϰāĻžāϝāĻŧāĻļāχ āϜāĻŋāĻœā§āĻžāĻžāϏāĻŋāϤ āĻĒā§āϰāĻļā§āύāĻžāĻŦāϞ⧀

āĻ“āĻ­āĻžāϰāĻ­āĻŋāω

In OpenAI's 2023 study 'Let's Verify Step by Step', a step-level verifier picked correct answers to 78.2% of held-out MATH problems versus 72.4% for an answer-only verifier, making process reward models a central idea in how language models are trained and steered to reason.

āĻŽā§‚āϞ āĻŸā§‡āĻ•āĻ“āϝāĻŧ⧇

  1. Process supervision labels each reasoning step, while outcome supervision labels only whether the final answer is correct.
  2. In Let's Verify Step by Step, a process reward model solved 78.2% of 500 held-out MATH problems via best-of-1860 selection, versus 72.4% for an outcome model.
  3. PRM800K contains about 800,000 human step labels across 75,000 solutions to 12,000 MATH problems.
  4. A 2022 GSM8K study by Uesato and colleagues found outcome feedback matched final-answer accuracy, but process feedback cut reasoning errors from 14.0% to 3.4%.
  5. Math-Shepherd generates step labels automatically by completing partial solutions many times and checking how often they reach the correct answer.
  6. DeepSeek-R1's authors reported that process reward models invited reward hacking and added too much overhead for their large-scale reinforcement learning.

āĻ—āĻ­ā§€āϰ āĻĄā§āĻŦ

When a language model solves a math problem, it usually writes a chain of steps before giving an answer. There are two ways to give feedback on such a solution. Outcome supervision checks only the final answer: right or wrong. Process supervision checks each intermediate step: is this line correct, and does it move toward a solution? The feedback is used to train a reward model, a separate network that scores solutions. A reward model trained on step labels is called a process reward model (PRM); one trained on final-answer labels is an outcome reward model (ORM). Reward models can then be used to pick the best of several candidate answers, a verifier role, or to provide training signals for reinforcement learning.

The motivation is easy to see with an example. A model can reach the correct number through a wrong argument, such as two errors that cancel, or a lucky guess. Outcome supervision rewards that solution anyway. It also gives only one bit of information for a long solution, so the learner cannot tell which step went wrong, a problem known as credit assignment. Process supervision points to the exact step that failed.

An early large comparison came from Jonathan Uesato, Geoffrey Irving, Irina Higgins and colleagues in a paper posted November 25, 2022, using GSM8K, a set of grade-school word problems. They found that outcome-based supervision gave similar final-answer error rates with fewer labels. However, getting the reasoning itself right needed process-based feedback or reward models that imitate it. Their best setup cut final-answer error from 16.8% to 12.7%, and cut reasoning errors among solutions with correct final answers from 14.0% to 3.4%.

OpenAI's 'Let's Verify Step by Step', by Hunter Lightman and colleagues and posted May 31, 2023, scaled the question up. The team fine-tuned models from base GPT-4, trained a generator to write solutions one step per line, and had human labelers rate each step of solutions to problems from MATH, a benchmark of competition mathematics. A step was positive if correct and reasonable, negative if incorrect or unreasonable, and neutral if ambiguous. Labeling stopped at the first mistake, so for a wrong solution the process label adds one fact the outcome label lacks: where the error is. The resulting dataset, PRM800K, contains about 800,000 step labels across 75,000 solutions to 12,000 problems.

The headline result comes from best-of-N search. For each of 500 held-out MATH test problems, the generator produced up to 1,860 solutions; each reward model picked its top-ranked solution, which was graded on its final answer. The PRM chose a correct solution 78.2% of the time, the ORM 72.4%, and majority voting, which picks the most common final answer, 69.6%. The gap grew as more candidates were sampled. On 224 newer AP Calculus, Chemistry, Physics and AMC questions released after the pretraining data was gathered, the PRM scored 72.9% in aggregate versus 63.8% for the ORM and 61.3% for majority voting. The authors also reported that active learning, showing labelers convincing wrong-answer solutions that fooled the current PRM, was about 2.6 times more data-efficient than labeling solutions at random.

Human labels are expensive, so later work tried to generate step labels automatically. Math-Shepherd, posted December 14, 2023 by Peiyi Wang and colleagues, labels a step by letting a model, called a completer, finish the solution from that step several times. A step that often leads to the right answer is treated as good. Trained this way without human annotation, Math-Shepherd raised Mistral-7B from 28.6% to 33.0% on MATH through step-level reinforcement learning, and to 43.5% when also used to verify candidate answers.

Process supervision has clear limitations. The DeepSeek-R1 paper, posted January 22, 2025, lists PRMs among its unsuccessful attempts and names three problems: it is hard to define what counts as a step in general reasoning, hard to judge whether an intermediate step is correct, and a learned reward model invites reward hacking, where the policy learns to please the verifier rather than be right. The authors concluded that PRMs work well for reranking and guided search but did not justify their added compute in large-scale reinforcement learning, and they trained with rule-based answer checks instead. Lightman and colleagues also note open issues: they could not rule out some test-set contamination, they studied only math, and their PRM is itself a model that can be wrong. Automatic labels like Math-Shepherd's contain noise, and the paper lists the cost of the completion process as a limitation.

In practice, process supervision needs three things: problems with checkable answers, solutions split into clear steps, and a labeling budget, human or computational. Using a PRM at inference time also costs compute, since best-of-N requires generating and scoring many candidates. Lightman and colleagues argued the method carries a negative alignment tax: it improved accuracy while rewarding reasoning that humans endorse, which they view as a safety benefit. Whether that holds outside math is, in their words, unknown.

04āĻ•āĻžāĻœā§‡āϰ āωāĻĻāĻžāĻšāϰāĻŖ

Scoring two solutions with step-level probabilities

  1. Take a problem with a known answer, such as solving 3x + 5 = 20 for x, and write two four-step solutions: one correct, one that divides wrongly in step 3 but still states x = 5.

  2. Assign each step a probability of being correct, as a PRM would: for the correct solution use 0.98, 0.97, 0.96 and 0.99.

  3. For the flawed solution, use 0.98, 0.97, 0.20 and 0.95 so the bad step gets a low score.

  4. Compute each solution score as the product of its step probabilities: about 0.90 for the correct solution and about 0.18 for the flawed one.

  5. Note what an outcome-only check would do: both solutions state x = 5, so both would be marked correct.

  6. Compare with how PRM800K would label the flawed solution: positive labels for steps 1 and 2, a negative label at step 3, and no labels after it.

āĻāϟāĻž āĻ•āĻŋ āĻĻ⧇āĻ–āĻžāϝāĻŧ

The step-level scores separate a sound solution from a lucky one that outcome checking cannot tell apart. Check the arithmetic by multiplying the four probabilities yourself, and confirm that changing only the weakest step has the biggest effect on the product.

āĻŦāĻžāĻ¸ā§āϤāĻŦ-āĻŦāĻŋāĻļā§āĻŦ āĻŦāĻžāĻ¸ā§āϤāĻŦāĻžāϝāĻŧāύ

Best-of-N reranking: a model writes many candidate solutions, and a process reward model picks the one whose steps all look correct, the evaluation setup used in Let's Verify Step by Step.

Studying human step labels in PRM800K, OpenAI's MIT-licensed release of 800,000 step-level ratings marking each solution step as positive, negative or neutral.

Generating step labels without people, as Math-Shepherd (2023) does by letting a model finish each partial solution several times and checking how often it reaches the right answer.

Reinforcement learning with step rewards: Math-Shepherd reported step-by-step PPO raising Mistral-7B from 77.9% to 84.1% on the GSM8K grade-school math benchmark.

Auditing tutoring or homework-help systems by flagging the first incorrect line of a worked solution, rather than only marking the final answer wrong.

āĻ—āĻŖāĻŋāϤ āϝ⧁āĻ•ā§āϤāĻŋāϰ āϜāĻ¨ā§āϝ āĻĒā§āϰāĻ•ā§āϰāĻŋāϝāĻŧāĻž āϤāĻ¤ā§āĻ¤ā§āĻŦāĻžāĻŦāϧāĻžāύ⧇āϰ āĻ­āĻŦāĻŋāĻˇā§āϝāϤ

The research direction has split. One line, represented by Lightman and colleagues in 2023 and Math-Shepherd in 2023 to 2024, keeps improving step-level verifiers and cheaper ways to label them. Another, represented by DeepSeek-R1 in January 2025, trains reasoning models with simple rule-based rewards on final answers and reports that PRMs were not worth their overhead at scale, while still useful for reranking and guided search. Open questions include how to define steps in domains without line-by-line structure, such as proofs, code or scientific argument; how to keep a learned verifier from being gamed during long reinforcement learning runs; and how to get reliable step labels without paying for expert annotators. Lightman and colleagues explicitly called for testing process supervision outside mathematics. Another open issue is evaluation itself: results on MATH may be affected by contamination, which is why the 2023 study added newly released exam questions. Process supervision remains relevant to safety research, because rewarding an inspectable reasoning process rather than just results is one proposed way to keep model behavior aligned with human judgment, but how far that benefit extends is not settled.

āĻāϟāĻžāϕ⧇ āĻŦāĻžāĻ¸ā§āϤāĻŦ⧇ āĻĒā§āϰāϝāĻŧā§‹āĻ— āĻ•āϰāĻž āĻšāĻšā§āϛ⧇

āϕ⧇āύ āĻāϟāĻž āϗ⧁āϰ⧁āĻ¤ā§āĻŦāĻĒā§‚āĻ°ā§āĻŖ

  • āĻ—āϤāĻŋ āĻāĻŦāĻ‚ āĻ¸ā§āϕ⧇āϞ. āĻ­āĻžāώāĻžāϰ āĻ•āĻ°ā§āĻŽāĻĒā§āϰāĻŦāĻžāĻš āϧāĻžāϰāĻžāĻŦāĻžāĻšāĻŋāĻ•āϤāĻžāϕ⧇ āĻ¤ā§āϝāĻžāĻ— āύāĻž āĻ•āϰ⧇ āĻĻā§āϰ⧁āϤ āĻ…āĻ—ā§āϰāϏāϰ āĻšāϤ⧇ āĻĒāĻžāϰ⧇āĨ¤
  • āĻ…ā§āϝāĻžāĻ•ā§āϏ⧇āϏ āĻāĻŦāĻ‚ āĻĒ⧌āρāĻ›āĻžāύ⧋. āĻāϟāĻŋ āĻ­āĻžāώāĻž āĻāĻŦāĻ‚ āϝ⧋āĻ—āĻžāϝ⧋āĻ— āĻļ⧈āϞ⧀ āϜ⧁āĻĄāĻŧ⧇ āĻ…ā§āϝāĻžāĻ•ā§āϏ⧇āϏ āĻĒā§āϰāϏāĻžāϰāĻŋāϤ āĻ•āϰ⧇āĨ¤
  • āϏ⧁āĻ¸ā§āĻĒāĻˇā§āϟ āϏāĻŋāĻĻā§āϧāĻžāĻ¨ā§āϤ. āĻ…āĻŸā§‹āĻŽā§‡āĻļāύ āĻĒ⧁āύāϰāĻžāĻŦ⧃āĻ¤ā§āϤāĻŋ āĻĒāϰāĻŋāϚāĻžāϞāύāĻž āĻ•āϰāĻžāϰ āϏāĻŽāϝāĻŧ āĻĻāϞāϗ⧁āϞāĻŋ āĻŦāĻŋāϚāĻžāϰ⧇ āφāϰāĻ“ āĻŦ⧇āĻļāĻŋ āϏāĻŽāϝāĻŧ āĻŦā§āϝāϝāĻŧ āĻ•āϰāϤ⧇ āĻĒāĻžāϰ⧇āĨ¤

āϜāĻ¨ā§āϝ āϏāϤāĻ°ā§āĻ•

  • āĻšā§āϝāĻžāϞ⧁āϏāĻŋāύ⧇āĻŸā§‡āĻĄ āĻĢā§āϝāĻžāĻ•ā§āϟāϗ⧁āϞāĻŋ āĻļāĻžāĻ¨ā§āϤāĻ­āĻžāĻŦ⧇ āϰāĻŋāĻĒā§‹āĻ°ā§āϟ, āϏāĻŽāĻ°ā§āĻĨāύ āĻĒā§āϰāĻŦāĻžāĻš, āĻŦāĻž āĻ—āĻŦ⧇āώāĻŖāĻž āφāωāϟāĻĒ⧁āϟ āĻĒā§āϰāĻŦ⧇āĻļ āĻ•āϰāϤ⧇ āĻĒāĻžāϰ⧇āĨ¤
  • āĻĒā§āϰāĻŽā§āĻĒāϟ āϏāĻ‚āĻŦ⧇āĻĻāύāĻļā§€āϞāϤāĻž āĻ…āύ⧁āϰ⧂āĻĒ āĻ…āύ⧁āϰ⧋āϧ āϜ⧁āĻĄāĻŧ⧇ āĻ…āϏāĻžāĻŽāĻžā§āϜāĻ¸ā§āϝāĻĒā§‚āĻ°ā§āĻŖ āĻĢāϞāĻžāĻĢāϞ āϤ⧈āϰāĻŋ āĻ•āϰāϤ⧇ āĻĒāĻžāϰ⧇āĨ¤
  • āĻ…ā§āϝāĻžāĻ•ā§āϏ⧇āϏ āĻ•āĻ¨ā§āĻŸā§āϰ⧋āϞ āĻĻ⧁āĻ°ā§āĻŦāϞ āĻšāϞ⧇ āϏāĻ‚āĻŦ⧇āĻĻāύāĻļā§€āϞ āĻĒāĻžāĻ ā§āϝ āĻĄā§‡āϟāĻž āωāĻ¨ā§āĻŽā§āĻ•ā§āϤ āĻšāϤ⧇ āĻĒāĻžāϰ⧇āĨ¤

āĻĒāϰāĻŦāĻ°ā§āϤ⧀ āĻĒāĻĻāĻ•ā§āώ⧇āĻĒ

  1. āϰ⧋āϞāφāωāϟ āĻ•āϰāĻžāϰ āφāϗ⧇ āφāωāϟāĻĒ⧁āϟ āĻĢāĻ°ā§āĻŽā§āϝāĻžāϟ, āĻŸā§‹āύ āĻāĻŦāĻ‚ āĻŽāĻžāύ⧇āϰ āĻŽāĻžāύ āύāĻŋāĻ°ā§āϧāĻžāϰāĻŖ āĻ•āϰ⧁āύāĨ¤
  2. āϝāĻ–āύāχ āύāĻŋāĻ°ā§āϭ⧁āϞāϤāĻž āϗ⧁āϰ⧁āĻ¤ā§āĻŦāĻĒā§‚āĻ°ā§āĻŖ āϤāĻ–āύ āĻŦāĻŋāĻļā§āĻŦāĻ¸ā§āϤ āωāĻ¤ā§āϏ⧇āϰ āϏāĻžāĻĨ⧇ āĻ—ā§āϰāĻžāωāĻ¨ā§āĻĄ āĻĒā§āϰāϤāĻŋāĻ•ā§āϰāĻŋāϝāĻŧāĻžāĨ¤
  3. āωāĻšā§āϚ-āĻ¸ā§āĻŸā§‡āϕ⧇āϰ āφāωāϟāĻĒ⧁āϟāϗ⧁āϞāĻŋāϰ āϜāĻ¨ā§āϝ āĻāĻ•āϟāĻŋ āĻŽāĻžāύāĻŦ āĻĒāĻ°ā§āϝāĻžāϞ⧋āϚāύāĻž āĻšā§‡āĻ•āĻĒāϝāĻŧ⧇āĻ¨ā§āϟ āϰāĻžāϖ⧁āύāĨ¤
  4. āĻŦā§āϝāĻ°ā§āĻĨāϤāĻžāϰ āύāĻŋāĻĻāĻ°ā§āĻļāύāϗ⧁āϞāĻŋ āĻŸā§āĻ°ā§āϝāĻžāĻ• āĻ•āϰ⧁āύ āĻāĻŦāĻ‚ āĻĒā§āϰāĻŽā§āĻĒāϟ āĻŦāĻž āĻ“āϝāĻŧāĻžāĻ°ā§āĻ•āĻĢā§āϞ⧋āϗ⧁āϞāĻŋāϕ⧇ āύāĻŋāϝāĻŧāĻŽāĻŋāϤāĻ­āĻžāĻŦ⧇ āĻĒ⧁āύāϰāĻžāϝāĻŧ āĻĒā§āϰāĻļāĻŋāĻ•ā§āώāĻŖ āĻĻāĻŋāύāĨ¤

āϏ⧂āĻ¤ā§āϰ āĻāĻŦāĻ‚ āφāϰāĻ“ āĻĒāĻĄāĻŧ⧁āύ

  1. arXiv (OpenAI)Let's Verify Step by Step
  2. OpenAI (GitHub)PRM800K: A Process Supervision Dataset
  3. arXivSolving math word problems with process- and outcome-based feedback
  4. arXivMath-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
  5. arXiv (DeepSeek-AI)DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

āĻ…āĻ¨ā§āĻŦ⧇āώāĻŖ āϚāĻžāϞāĻŋāϝāĻŧ⧇ āϝāĻžāύ

āĻĢā§āϰāĻŋ āύāĻŋāωāϜāϞ⧇āϟāĻžāϰ

āĻĒā§āϰāϤāĻŋāĻĻāĻŋāύ⧇āϰ āĻāφāχ āĻŦā§āϰāĻŋāĻĢāĻŋāĻ‚ āĻĒāĻžāύ

āĻĒā§āϰāϤāĻŋ āϏāĻĒā§āϤāĻžāĻšā§‡āϰ āĻ•āĻžāĻœā§‡āϰ āĻĻāĻŋāύ āϏāĻ•āĻžāϞ⧇ āϤāĻŋāύāϟāĻŋ āϝāĻžāϚāĻžāχāĻ•ā§ƒāϤ āĻāφāχ āĻ¸ā§āĻŸā§‹āϰāĻŋ, āϏāĻšāϜ āχāĻ‚āϰ⧇āϜāĻŋāϤ⧇ āϞ⧇āĻ–āĻžāĨ¤ āϚāĻŋāϰāĻ•āĻžāϞ⧇āϰ āϜāĻ¨ā§āϝ āĻĢā§āϰāĻŋ, āϕ⧋āύ⧋ āĻŦāĻŋāĻœā§āĻžāĻžāĻĒāύ āύ⧇āχāĨ¤

āĻĒā§āϰāϤāĻŋ āϏāĻĒā§āϤāĻžāĻšā§‡āϰ āĻ•āĻžāĻœā§‡āϰ āĻĻāĻŋāύ⧇ āĻāĻ•āϟāĻŋ āχāĻŽā§‡āϞāĨ¤ āĻāĻ• āĻ•ā§āϞāĻŋāϕ⧇āχ āφāύāϏāĻžāĻŦāĻ¸ā§āĻ•ā§āϰāĻžāχāĻŦ āĻ•āϰ⧁āύāĨ¤ āφāĻŽāϰāĻž āĻ•āĻ–āύ⧋āχ āφāĻĒāύāĻžāϰ āĻ āĻŋāĻ•āĻžāύāĻž āĻŦāĻŋāĻ•ā§āϰāĻŋ āĻŦāĻž āĻļā§‡ā§ŸāĻžāϰ āĻ•āϰāĻŋ āύāĻžāĨ¤

āύāĻŋāĻœā§‡āϕ⧇ āĻĒāϰ⧀āĻ•ā§āώāĻž āĻ•āϰ⧁āύ

Take the Process Supervision for Math Reasoning quiz

āĻĒā§āϰāϤāĻŋāϟāĻŋ āωāĻ¤ā§āϤāϰ⧇āϰ āϤāĻžā§ŽāĻ•ā§āώāĻŖāĻŋāĻ• āĻĢāĻŋāĻĄāĻŦā§āϝāĻžāĻ•, āĻāĻŦāĻ‚ āϕ⧋āĻ°ā§āϏ āĻĒāĻžāϏ āĻ•āϰāĻžāϰ āĻĒāϰ āϝāĻžāϚāĻžāχāϝ⧋āĻ—ā§āϝ āφāχāĻĄāĻŋ āϏāĻš āĻāĻ•āϟāĻŋ āĻļā§‡ā§ŸāĻžāϰāϝ⧋āĻ—ā§āϝ āϏāĻžāĻ°ā§āϟāĻŋāĻĢāĻŋāϕ⧇āϟāĨ¤

āϕ⧁āχāϜ āĻļ⧁āϰ⧁ āĻ•āϰ⧁āύ

āĻĢā§āϰāĻŋ āĻāφāχ āĻļāĻŋāĻ•ā§āώāĻžāϕ⧇ āϏāĻŽāĻ°ā§āĻĨāύ āĻ•āϰ⧁āύāĨ¤ AI Understanding āĻāĻ•āϟāĻŋ 501(c)(3) āĻ…āϞāĻžāĻ­āϜāύāĻ• āĻĒā§āϰāϤāĻŋāĻˇā§āĻ āĻžāύ — āϕ⧋āύ⧋ āĻŦāĻŋāĻœā§āĻžāĻžāĻĒāύ āύ⧇āχ, āϕ⧋āύ⧋ āĻĒ⧇āĻ“ā§ŸāĻžāϞ āύ⧇āχ, āĻ•āĻ–āύ⧋āχ āύāĻžāĨ¤ āĻāĻ•āϟāĻŋ āĻĻāĻžāύ āĻ•āϰ⧁āύ

āĻĒā§āϰāĻžāϝāĻŧāĻļāχ āϜāĻŋāĻœā§āĻžāĻžāϏāĻŋāϤ āĻĒā§āϰāĻļā§āύāĻžāĻŦāϞ⧀

āĻ—āĻŖāĻŋāϤ āϝ⧁āĻ•ā§āϤāĻŋāϰ āϜāĻ¨ā§āϝ āĻĒā§āϰāĻ•ā§āϰāĻŋāϝāĻŧāĻž āϤāĻ¤ā§āĻ¤ā§āĻŦāĻžāĻŦāϧāĻžāύ āĻ•āĻŋ?

Process supervision trains a reward model to judge each step of a worked solution, instead of only checking the final answer. In OpenAI's 2023 study 'Let's Verify Step by Step', a step-level verifier picked correct answers to 78.2% of held-out MATH problems versus 72.4% for an answer-only verifier, making process reward models a central idea in how language models are trained and steered to reason.

āĻĢāϞāĻžāĻĢāϞ āϤāĻ¤ā§āĻ¤ā§āĻŦāĻžāĻŦāϧāĻžāύ⧇āϰ āϤ⧁āϞāύāĻžāϝāĻŧ āĻĒā§āϰāĻ•ā§āϰāĻŋāϝāĻŧāĻž āϤāĻ¤ā§āĻ¤ā§āĻŦāĻžāĻŦāϧāĻžāύ āϕ⧋āĻĨāĻžāϝāĻŧ āĻĒā§āϰāϤāĻŋāĻ•ā§āϰāĻŋāϝāĻŧāĻž āĻĒā§āϰāĻĻāĻžāύ āĻ•āϰ⧇?

āĻĒāĻžāĻ°ā§āĻĨāĻ•ā§āϝ āĻšāϞ āĻĒā§āϰāϤāĻŋāĻ•ā§āϰāĻŋāϝāĻŧāĻžāϰ āĻ…āĻŦāĻ¸ā§āĻĨāĻžāύ; āϕ⧋āύ āĻĒāĻĻā§āϧāϤāĻŋāχ āύāĻŋāϖ⧁āρāϤ āĻŦāĻŋāϚāĻžāϰ⧇āϰ āύāĻŋāĻļā§āϚāϝāĻŧāϤāĻž āĻĻ⧇āϝāĻŧ āύāĻžāĨ¤

āϕ⧇āύ āĻļ⧁āϧ⧁āĻŽāĻžāĻ¤ā§āϰ āĻĢāϞāĻžāĻĢāϞ-āϤāĻ¤ā§āĻ¤ā§āĻŦāĻžāĻŦāϧāĻžāύ āĻ—āĻŖāĻŋāϤ āϏāĻŽāĻ¸ā§āϝāĻžāϰ āϜāĻ¨ā§āϝ āĻŦāĻŋāĻ­ā§āϰāĻžāĻ¨ā§āϤāĻŋāĻ•āϰ āĻšāϤ⧇ āĻĒāĻžāϰ⧇?

āĻļ⧁āϧ⧁āĻŽāĻžāĻ¤ā§āϰ āĻšā§‚āĻĄāĻŧāĻžāĻ¨ā§āϤ āωāĻ¤ā§āϤāϰāϕ⧇ āĻĒ⧁āϰāĻ¸ā§āĻ•ā§ƒāϤ āĻ•āϰāĻž 'āĻ­āĻžāĻ—ā§āϝāĻŦāĻžāύ' āϏāĻŽāĻžāϧāĻžāύ⧇āϰ āĻ•ā§ƒāϤāĻŋāĻ¤ā§āĻŦ āĻĻ⧇āϝāĻŧ āϝ⧇āĻ–āĻžāύ⧇ āϭ⧁āϞ āĻŽāĻ§ā§āϝāĻŦāĻ°ā§āϤ⧀ āĻĒāĻĻāĻ•ā§āώ⧇āĻĒāϗ⧁āϞāĻŋ āĻ•āĻžāĻ•āϤāĻžāϞ⧀āϝāĻŧāĻ­āĻžāĻŦ⧇ āϏāĻ āĻŋāĻ• āĻĢāϞāĻžāĻĢāϞ āĻĻ⧇āϝāĻŧāĨ¤

OpenAI āĻ•āĻŋ 'āϞ⧇āϟāϏ āϭ⧇āϰāĻŋāĻĢāĻžāχ āĻ¸ā§āĻŸā§‡āĻĒ āĻŦāĻžāχ āĻ¸ā§āĻŸā§‡āĻĒ' āĻāϰ āϏāĻžāĻĨ⧇ āϰāĻŋāϞāĻŋāϜ āĻ•āϰ⧇āϛ⧇?

PRM800K-āĻ āĻŽā§‹āϟāĻžāĻŽā§āϟāĻŋ 800,000 āϟāĻŋ āĻŽāĻžāύāĻŦ āĻŸā§€āĻ•āĻž āϰāϝāĻŧ⧇āϛ⧇ āϝāĻž āĻĒ⧃āĻĨāĻ• āϝ⧁āĻ•ā§āϤāĻŋāϰ āĻĒāĻĻāĻ•ā§āώ⧇āĻĒāϗ⧁āϞāĻŋāϕ⧇ āϏāĻ āĻŋāĻ• āĻŦāĻž āϭ⧁āϞ āĻšāĻŋāϏāĻžāĻŦ⧇ āϚāĻŋāĻšā§āύāĻŋāϤ āĻ•āϰ⧇⧎

āĻāĻ•āϟāĻŋ āĻĒā§āϰāĻ•ā§āϰāĻŋāϝāĻŧāĻž āĻĒ⧁āϰāĻˇā§āĻ•āĻžāϰ āĻŽāĻĄā§‡āϞ āϏāĻš āϏ⧇āϰāĻž-āĻ…āĻĢ-N āĻ—āĻŖāĻŋāϤ āύāĻŋāĻ°ā§āĻŦāĻžāϚāύ⧇, āϝāĻžāϚāĻžāχāĻ•āĻžāϰ⧀ āϕ⧀ āĻ•āϰ⧇?

āĻāϟāĻŋ āωāĻĒāϞāĻŦā§āϧ āĻĒā§āϰāĻžāĻ°ā§āĻĨā§€āĻĻ⧇āϰ āĻŽāĻ§ā§āϝ⧇ āύāĻŋāĻ°ā§āĻŦāĻžāϚāύ āĻ•āϰ⧇; āϝāĻžāϚāĻžāχāĻ•āĻžāϰ⧀ āĻ¤ā§āϰ⧁āϟāĻŋ āĻāĻŦāĻ‚ āĻĒā§āϰāĻžāĻ°ā§āĻĨā§€āϰ āĻ•āĻ­āĻžāϰ⧇āϜ āĻāĻ–āύāĻ“ āϗ⧁āϰ⧁āĻ¤ā§āĻŦāĻĒā§‚āĻ°ā§āĻŖāĨ¤

āϕ⧋āύ āĻĒāĻĻā§āϧāϤāĻŋ āĻŦā§āϝāϝāĻŧāĻŦāĻšā§āϞ āĻŽāĻžāύāĻŦ āϧāĻžāĻĒ āϞ⧇āĻŦ⧇āϞ āϜāĻ¨ā§āϝ āĻĒā§āϰāϝāĻŧā§‹āϜāύ⧀āϝāĻŧāϤāĻž āĻšā§āϰāĻžāϏ?

āĻŽā§āϝāĻžāĻĨ-āĻļ⧇āĻĢāĻžāĻ°ā§āĻĄ āĻŽā§āϝāĻžāύ⧁āϝāĻŧāĻžāϞ āϞ⧇āĻŦ⧇āϞāĻŋāĻ‚ āĻāĻĄāĻŧāĻŋāϝāĻŧ⧇, āĻ•āϤ āϘāύ āϘāύ āϏāĻ āĻŋāĻ• āωāĻ¤ā§āϤāϰ⧇ āĻĒ⧌āρāĻ›āĻžāϝāĻŧ āϤāĻž āĻĒāϰāĻŋāĻŽāĻžāĻĒ āĻ•āϰ⧇ āϏāĻŽāĻžāĻĒā§āϤāĻŋāϗ⧁āϞāĻŋ āϰ⧋āϞ āφāωāϟ āĻ•āϰ⧇ āϧāĻžāĻĒ⧇āϰ āϏāĻ āĻŋāĻ•āϤāĻž āĻ…āύ⧁āĻŽāĻžāύ āĻ•āϰ⧇āĨ¤

āĻļāĻŋāĻ–āϤ⧇ āĻĨāĻžāϕ⧁āύ

āϏāĻŽā§āĻĒāĻ°ā§āĻ•āĻŋāϤ āĻ—āĻžāχāĻĄ

āĻāχ āĻŦāĻŋāώāϝāĻŧ⧇āϰ āϜāĻ¨ā§āϝ āĻŦāĻžāĻ›āĻžāχ āĻ•āϰāĻž āφāϰāĻ“ āĻ—āĻžāχāĻĄ

āĻĒāϰāĻŦāĻ°ā§āϤ⧀ āφāĻĒāĻĒāϰāĻŦāĻ°ā§āϤ⧀ āĻ—āĻžāχāĻĄ

āĻšā§‡āχāύ-āĻ…āĻĢ-āĻĨāϟ āϰāĻŋāϜāύāĻŋāĻ‚

āĻ­āĻžāώāĻž āĻāφāχ

āĻāĻ•āϟāĻŋ āĻŦāĻŋāύāĻžāĻŽā§‚āĻ˛ā§āϝ⧇ AI āĻļāĻ‚āϏāĻžāĻĒāĻ¤ā§āϰ āĻĒāĻžāύāĻāĻ•āϟāĻŋ āϏāĻ‚āĻ•ā§āώāĻŋāĻĒā§āϤ āĻŦāĻŋāύāĻžāĻŽā§‚āĻ˛ā§āϝ⧇ āϕ⧋āĻ°ā§āϏ āĻāĻŦāĻ‚ āĻĒāϰ⧀āĻ•ā§āώāĻž. āϕ⧋āύ⧋ āĻ–āϰāϚ āύ⧇āχ, āϕ⧋āύ⧋ āĻ•āĻžāĻ°ā§āĻĄ āύ⧇āχāĨ¤ āĻĒā§āϰāĻžāϝāĻŧ 2.5 āϘāĻ¨ā§āϟāĻžāĨ¤āĻŦāĻŋāύāĻžāĻŽā§‚āĻ˛ā§āϝ⧇ āĻļ⧁āϰ⧁ āĻ•āϰ⧁āύ →