Reinforcement Fine-Tuning with Graders

Standard supervised fine-tuning shows the model an input and the exact output you want, and trains it to imitate. That works when you can write good target outputs, but it teaches the surface form of answers rather than the process of reaching them. Reinforcement fine-tuning works differently. For each training prompt, the model samples one or more candidate answers. A grader assigns each a score, often between 0 and 1. The training algorithm then adjusts the model so that answers scoring above average become more likely and those below average less likely. Over many rounds, the model keeps the reasoning patterns that earn reward. Graders come in a few kinds. Exact or string-match graders check whether a final answer equals a reference. Programmatic graders run code, such as unit tests or a numerical tolerance check. Model-based graders use another language model with a rubric to judge qualities that are hard to check mechanically. Graders can also give partial credit, which gives the model a smoother signal. RFT suits tasks with verifiable or reliably gradable answers where experts agree on what is correct: classification with defined labels, maths, code, structured extraction, and some specialised judgement tasks. It is a poor fit where quality is subjective and graders disagree. OpenAI previewed a reinforcement fine-tuning service in December 2024, and reasoning models more broadly have been trained with reinforcement learning on verifiable rewards. The central failure mode is reward hacking. The model optimises the grader, not your intent. If a grader only checks the final line, the model may produce a correct-looking final line with broken reasoning. If a model grader likes length or confident wording, outputs drift toward those traits. A common misconception is that a higher training reward means a better model; it only means a better score from that grader.