Process Supervision for Math Reasoning

A correct final number does not establish that every step leading to it was sound. Outcome feedback can therefore credit a solution containing errors that happen to cancel, whereas process feedback provides information about intermediate steps. Let’s Verify Step by Step investigates this distinction on math problems and releases PRM800K, containing about 800,000 step-level human feedback labels. Its reported results are tied to the studied models, candidate sampling, and a representative subset of the MATH test set. They are not a universal accuracy guarantee for all process reward models. At inference, a verifier can score multiple sampled solutions and select one according to an explicitly defined rule. The cited paper studies products and minimums of step scores, along with how neutral steps are treated. Its main reported rule uses the product of step-level scores and treats neutral steps as positive. Multiplication can favor shorter solutions even when their individual step scores are comparable, so aggregation is a modeling choice rather than a trivial detail. For scores 0.9 and 0.8, the product is 0.72 and the minimum is 0.8. Those different summaries should not be confused. The paper also investigates active learning to select informative labeling examples. Math-Shepherd instead constructs process supervision automatically and studies both verification and step-by-step reinforcement learning. Automatically estimated labels still need evaluation. A low verifier score can direct attention to a suspicious line, but does not establish that this is exactly the first actual mistake.