Xiga xigaHagaha xiga
Continued Pretraining vs Fine-Tuning
Farsamo
HAGAHA Farsamada
Reinforcement Fine-Tuning with Graders
Reinforcement fine-tuning (RFT) trains a model by having it generate answers, scoring those answers with a grader, and updating the model to make high-scoring answers more likely, rather than teaching it to copy example outputs.
Boggaan4 daqiiqo akhri
Dulmar
It matters because it can improve reasoning on tasks where a correct answer is easy to check but hard to demonstrate step by step. Its main risk is reward hacking, where the model learns to please the grader instead of solving the task.
quusid qoto dheer
Standard supervised fine-tuning shows the model an input and the exact output you want, and trains it to imitate. That works when you can write good target outputs, but it teaches the surface form of answers rather than the process of reaching them. Reinforcement fine-tuning works differently. For each training prompt, the model samples one or more candidate answers. A grader assigns each a score, often between 0 and 1. The training algorithm then adjusts the model so that answers scoring above average become more likely and those below average less likely. Over many rounds, the model keeps the reasoning patterns that earn reward. Graders come in a few kinds. Exact or string-match graders check whether a final answer equals a reference. Programmatic graders run code, such as unit tests or a numerical tolerance check. Model-based graders use another language model with a rubric to judge qualities that are hard to check mechanically. Graders can also give partial credit, which gives the model a smoother signal. RFT suits tasks with verifiable or reliably gradable answers where experts agree on what is correct: classification with defined labels, maths, code, structured extraction, and some specialised judgement tasks. It is a poor fit where quality is subjective and graders disagree. OpenAI previewed a reinforcement fine-tuning service in December 2024, and reasoning models more broadly have been trained with reinforcement learning on verifiable rewards. The central failure mode is reward hacking. The model optimises the grader, not your intent. If a grader only checks the final line, the model may produce a correct-looking final line with broken reasoning. If a model grader likes length or confident wording, outputs drift toward those traits. A common misconception is that a higher training reward means a better model; it only means a better score from that grader.
Saamaynta Istiraatijiyadeed
Qiimaha iyo miisaaniyada
Go'aamada qaab-dhismeedku waxay horseedaan waxqabadka iyo kharashka hawlgalka sannadaha.
Go'aamo cad
Waxbarashada farsamada waxay ka caawisaa kooxaha inay doortaan xidhmo sax ah, ma aha oo kaliya kan ugu cusub.
Xakamaynta tayada
Doorashooyinka injineernimada ee wanaagsan waxay yareeyaan shilalka la isku halleyn karo ee wax soo saarka.
The Future of Reinforcement Fine-Tuning with Graders
Reinforcement learning on gradable tasks has become an important part of how reasoning models are trained, and hosted services are making RFT available to smaller teams. Progress is likely to depend less on the training algorithm and more on grader quality, since a model can only become as good as the signal it is optimised against. Research into more robust graders, multiple independent graders and better detection of reward hacking is active. For most organisations, the practical question will remain whether they can define correctness clearly enough to grade it.
Dhaqangelinta Adduunka-dhabta ah
A tax software team trains a model to classify expense items into the correct category code, with a grader that checks the predicted code against a labelled answer.
A company building a coding assistant rewards generated functions by running them against unit tests, so the reward reflects whether the code actually works.
A medical research group trains a model to rank likely genes given a list of symptoms, with a grader that gives partial credit when the correct gene appears near the top of the ranking.
A team uses a model-based grader to score customer email replies for accuracy and tone, then discovers the policy has learned to add flattering phrases the grader over-rewards.
Khatarta & Dariiqyada Ilaalada
Hagaajinta hal bartilmaameed waxay qarin kartaa daciifnimada nidaamka ballaaran.
Kaabayaasha dhaqaalaha iyo dayactirka inta badan waa la dhayalsadaa.
Nabadgelyada iyo daldaloolada u fiirsashada ayaa kori kara marka nidaamyadu noqdaan kuwo aad u adag.
Qorshe Hawleedka Dhaqangelinta
Qeex daahida, tayada, iyo bartilmaameedyada qiimaha ka hor inta aan la hirgelin.
Benchmark marka la eego culeyska dhabta ah iyo xaaladaha xogta.
La socodka qalabka khaladaadka, leexashada, iyo saamaynta isticmaalaha.
U diyaari dib-u-noqoshada iyo dariiqyada jawaab-celinta dhacdada ka hor inta aanad miisaan.
Sii wad Sahaminta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Reinforcement Fine-Tuning with Graders quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Su'aalaha soo noqnoqda
What is Reinforcement Fine-Tuning with Graders?
Reinforcement fine-tuning (RFT) trains a model by having it generate answers, scoring those answers with a grader, and updating the model to make high-scoring answers more likely, rather than teaching it to copy example outputs. It matters because it can improve reasoning on tasks where a correct answer is easy to check but hard to demonstrate step by step. Its main risk is reward hacking, where the model learns to please the grader instead of solving the task.
How does reinforcement fine-tuning differ from supervised fine-tuning?
Supervised fine-tuning imitates targets; RFT samples answers, scores them with a grader and reinforces high-scoring ones.
Which grader type would you use to check that generated code actually works?
Programmatic graders can execute code against tests, directly measuring whether it works.
Which task is the poorest fit for RFT according to the guide?
RFT needs a reliable grade. When experts disagree about what is correct, the reward signal is noisy.
In Reinforcement Fine-Tuning with Graders: what is reward hacking?
Reward hacking happens when optimisation exploits weaknesses in the grader, raising scores without improving true quality.
Why do prompts where every sample gets the same score provide little learning signal in group-relative methods like GRPO?
GRPO uses relative scores within a group as the advantage. If all scores match, there is no difference to learn from.
Sii wad waxbarashada
Tilmaamaha la xidhiidha
Tilmaamayaal badan ayaa loo doortay mawduucan