語言人工智慧指南

数学推理过程监督

流程監督會訓練一個獎勵模型來評估每一步的解題過程,而不是只檢查最終答案。

聆聽此指南AI 語音敘述 · ~7 min

本頁8 分鐘閱讀
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 以步驟層級機率為兩個解答打分
  5. 現實世界的實施
  6. 數學推理過程監督的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

在 OpenAI 2023 年的研究《Let's Verify Step by Step》中,步驟層級驗證器在保留的 MATH 題目中正確率為 78.2%,而僅答案驗證器為 72.4%,因此流程獎勵模型成為語言模型訓練與引導推理的核心概念。

重點摘要

  1. 流程監督會為每個推理步驟打標籤,而結果監督只標記最終答案是否正確。
  2. 在《Let's Verify Step by Step》中,流程獎勵模型在 500 題保留的 MATH 題目中以 1860 次選擇的最佳結果達到 78.2% 的正確率,結果模型則為 72.4%。
  3. PRM800K 包含約 80 萬筆人為步驟標籤,分佈於 7.5 萬個解答,涵蓋 1.2 萬題 MATH 題目。
  4. 2022 年 Uesato 等人在 GSM8K 上的研究發現,結果回饋與最終答案正確率相符,但流程回饋將推理錯誤率從 14.0% 降至 3.4%。
  5. Math‑Shepherd 透過多次完成部分解答並檢查其是否得到正確答案,自動產生步驟標籤。
  6. DeepSeek‑R1 的作者報告指出,流程獎勵模型會引發獎勵駭客行為,且在大規模強化學習中增加過多開銷。

深入探討

When a language model solves a math problem, it usually writes a chain of steps before giving an answer. There are two ways to give feedback on such a solution. Outcome supervision checks only the final answer: right or wrong. Process supervision checks each intermediate step: is this line correct, and does it move toward a solution? The feedback is used to train a reward model, a separate network that scores solutions. A reward model trained on step labels is called a process reward model (PRM); one trained on final-answer labels is an outcome reward model (ORM). Reward models can then be used to pick the best of several candidate answers, a verifier role, or to provide training signals for reinforcement learning.

The motivation is easy to see with an example. A model can reach the correct number through a wrong argument, such as two errors that cancel, or a lucky guess. Outcome supervision rewards that solution anyway. It also gives only one bit of information for a long solution, so the learner cannot tell which step went wrong, a problem known as credit assignment. Process supervision points to the exact step that failed.

An early large comparison came from Jonathan Uesato, Geoffrey Irving, Irina Higgins and colleagues in a paper posted November 25, 2022, using GSM8K, a set of grade-school word problems. They found that outcome-based supervision gave similar final-answer error rates with fewer labels. However, getting the reasoning itself right needed process-based feedback or reward models that imitate it. Their best setup cut final-answer error from 16.8% to 12.7%, and cut reasoning errors among solutions with correct final answers from 14.0% to 3.4%.

OpenAI's 'Let's Verify Step by Step', by Hunter Lightman and colleagues and posted May 31, 2023, scaled the question up. The team fine-tuned models from base GPT-4, trained a generator to write solutions one step per line, and had human labelers rate each step of solutions to problems from MATH, a benchmark of competition mathematics. A step was positive if correct and reasonable, negative if incorrect or unreasonable, and neutral if ambiguous. Labeling stopped at the first mistake, so for a wrong solution the process label adds one fact the outcome label lacks: where the error is. The resulting dataset, PRM800K, contains about 800,000 step labels across 75,000 solutions to 12,000 problems.

The headline result comes from best-of-N search. For each of 500 held-out MATH test problems, the generator produced up to 1,860 solutions; each reward model picked its top-ranked solution, which was graded on its final answer. The PRM chose a correct solution 78.2% of the time, the ORM 72.4%, and majority voting, which picks the most common final answer, 69.6%. The gap grew as more candidates were sampled. On 224 newer AP Calculus, Chemistry, Physics and AMC questions released after the pretraining data was gathered, the PRM scored 72.9% in aggregate versus 63.8% for the ORM and 61.3% for majority voting. The authors also reported that active learning, showing labelers convincing wrong-answer solutions that fooled the current PRM, was about 2.6 times more data-efficient than labeling solutions at random.

Human labels are expensive, so later work tried to generate step labels automatically. Math-Shepherd, posted December 14, 2023 by Peiyi Wang and colleagues, labels a step by letting a model, called a completer, finish the solution from that step several times. A step that often leads to the right answer is treated as good. Trained this way without human annotation, Math-Shepherd raised Mistral-7B from 28.6% to 33.0% on MATH through step-level reinforcement learning, and to 43.5% when also used to verify candidate answers.

Process supervision has clear limitations. The DeepSeek-R1 paper, posted January 22, 2025, lists PRMs among its unsuccessful attempts and names three problems: it is hard to define what counts as a step in general reasoning, hard to judge whether an intermediate step is correct, and a learned reward model invites reward hacking, where the policy learns to please the verifier rather than be right. The authors concluded that PRMs work well for reranking and guided search but did not justify their added compute in large-scale reinforcement learning, and they trained with rule-based answer checks instead. Lightman and colleagues also note open issues: they could not rule out some test-set contamination, they studied only math, and their PRM is itself a model that can be wrong. Automatic labels like Math-Shepherd's contain noise, and the paper lists the cost of the completion process as a limitation.

In practice, process supervision needs three things: problems with checkable answers, solutions split into clear steps, and a labeling budget, human or computational. Using a PRM at inference time also costs compute, since best-of-N requires generating and scoring many candidates. Lightman and colleagues argued the method carries a negative alignment tax: it improved accuracy while rewarding reasoning that humans endorse, which they view as a safety benefit. Whether that holds outside math is, in their words, unknown.

04工作範例

以步驟層級機率為兩個解答打分

  1. 以一個有已知答案的問題為例,例如求解 3x + 5 = 20 中的 x,並編寫兩個四步解決方案:一個正確的,另一個在第 3 步除法錯誤但仍得出 x = 5 的解決方案。

  2. 為每個步驟分配一個正確的機率,就像 PRM 所做的那樣:對於正確的解決方案,使用 0.98、0.97、0.96 和 0.99。

  3. 對於有缺陷的解決方案,使用 0.98、0.97、0.20 和 0.95,這樣錯誤的步驟就會得到較低的分數。

  4. 將每個解答的分數計算為其步驟機率的乘積:正確解答約為 0.90,而有瑕疵的解答約為 0.18。

  5. 請注意僅憑結果檢查會發生的情況:兩個解答都指出 x = 5,因此兩者都會被標記為正確。

  6. 與 PRM800K 標記該瑕疵解答的方式進行比較:第 1 步和第 2 步為正向標籤,第 3 步為負向標籤,之後則沒有標籤。

它顯示了什麼

步驟級分數能區分出紮實的解法與僅憑運氣的解法,而單靠檢查結果無法分辨兩者。請自行將四個機率相乘來驗算,並確認僅改變最薄弱的步驟對乘積的影響最大。

現實世界的實施

Best-of-N 重排序:模型會寫出許多候選解,然後由過程獎勵模型(Process Reward Model)挑選出所有步驟看起來都正確的解法,這正是《Let's Verify Step by Step》所使用的評估設置。

研究 PRM800K 中的人類步驟標籤,這是 OpenAI 以 MIT 授權釋出的 80 萬個步驟級評分,將每個解題步驟標記為正面、負面或中性。

在沒有人類參與的情況下生成步驟標籤,如同 Math-Shepherd (2023) 所做的那樣,讓模型多次完成每個部分解法,並檢查其達到正確答案的頻率。

使用步驟獎勵進行強化學習:Math-Shepherd 報告指出,逐步 PPO 將 Mistral-7B 在 GSM8K 小學數學基準測試中的準確率從 77.9% 提升至 84.1%。

稽核輔導或作業協助系統,方法是標記解題過程中第一個錯誤的行,而不僅僅是標記最終答案錯誤。

數學推理過程監督的未來

The research direction has split. One line, represented by Lightman and colleagues in 2023 and Math-Shepherd in 2023 to 2024, keeps improving step-level verifiers and cheaper ways to label them. Another, represented by DeepSeek-R1 in January 2025, trains reasoning models with simple rule-based rewards on final answers and reports that PRMs were not worth their overhead at scale, while still useful for reranking and guided search. Open questions include how to define steps in domains without line-by-line structure, such as proofs, code or scientific argument; how to keep a learned verifier from being gamed during long reinforcement learning runs; and how to get reliable step labels without paying for expert annotators. Lightman and colleagues explicitly called for testing process supervision outside mathematics. Another open issue is evaluation itself: results on MATH may be affected by contamination, which is why the 2023 study added newly released exam questions. Process supervision remains relevant to safety research, because rewarding an inspectable reasoning process rather than just results is one proposed way to keep model behavior aligned with human judgment, but how far that benefit extends is not settled.

付諸實踐

為什麼這很重要

  • 速度與規模. 語言工作流程可以在不犧牲一致性的情況下更快地移動。
  • 交通與覆蓋範圍. 它擴展了跨語言和溝通方式的訪問。
  • 更明確的決策. 團隊可以花更多時間進行判斷,而自動化則可以處理重複。

留意

  • 幻覺的事實可以悄悄地進入報告、支持流程或研究成果。
  • 及時的敏感性可能會在類似的請求中產生不一致的結果。
  • 如果存取控制薄弱,敏感文字資料可能會暴露。

後續步驟

  1. 在推出之前定義輸出格式、語氣和品質標準。
  2. 當準確性很重要時,請使用可信任來源進行地面回應。
  3. 為高風險輸出保留人工審查檢查點。
  4. 追蹤故障模式並定期重新訓練提示或工作流程。

資料來源與延伸閱讀

  1. arXiv (OpenAI)讓我們一步一步驗證
  2. OpenAI (GitHub)PRM800K:過程監督資料集
  3. arXiv利用過程與結果回饋解決數學應用題
  4. arXivMath-Shepherd:在沒有人類註解的情況下逐步驗證並強化大型語言模型
  5. arXiv (DeepSeek-AI)DeepSeek-R1:透過強化學習激勵大型語言模型的推理能力

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加數學推理的過程監督測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是數學推理的過程監督?

過程監督訓練獎勵模型來判斷解題的每個步驟,而不僅僅是檢查最終答案。在 OpenAI 2023 年的研究《Let's Verify Step by Step》中,步驟級驗證器在 MATH 測試集問題中選出正確答案的比例為 78.2%,而僅檢查答案的驗證器則為 72.4%,這使得過程獎勵模型成為訓練和引導語言模型進行推理的核心概念。

與結果監督相比,過程監督在何處提供回饋?

區別在於回饋的位置;這兩種方法都無法保證判斷絕對正確。

為什麼只注重結果的監督會對數學問題產生誤導?

僅獎勵最終答案會歸功於「幸運」的解決方案,其中不正確的中間步驟恰好產生了正確的結果。

OpenAI 與「讓我們逐步驗證」一起發布了什麼?

PRM800K 包含約 800,000 個人工註釋,將各個推理步驟標記為正確或錯誤。

在使用過程獎勵模型的 Best-of-N 數學選擇中,驗證器做了什麼?

它在現有的候選解中進行選擇;驗證器的錯誤率和候選解的覆蓋範圍仍然很重要。

什麼方法可以減少對昂貴的人工步驟標籤的需求?

Math-Shepherd 透過推出完成結果並測量正確答案的頻率來估計步驟的正確性,從而避免手動標記。