Rejection Sampling Fine-Tuning
Rejection Sampling Fine-Tuning (RFT) generates many candidate answers, keeps only the best-scoring ones, and retrains the model on those winners.
Overview
It matters because it offers much of RLHF's benefit using straightforward supervised learning instead of complex reinforcement learning.
Deep Dive
Rejection Sampling Fine-Tuning, sometimes called best-of-N fine-tuning, is a key ingredient in how models like Meta's Llama 2 and Llama 3 were aligned. The recipe is simple: for each prompt, sample several responses (say 4 to 64) from the current model, score each with a reward model or an automatic checker, then discard ('reject') all but the top-ranked outputs. The surviving high-quality samples become a fresh supervised fine-tuning dataset, and the model is trained on them with ordinary next-token loss. Repeating this loop iteratively nudges the model toward generating better answers on its own. Because the model learns from its own filtered outputs, RFT avoids the instability and tuning headaches of policy-gradient RL while still leveraging a reward signal.
Technical Insight
RFT exploits the fact that sampling many times and keeping the maximum-reward response approximates picking from a sharpened, higher-quality distribution. Training on those winners via standard cross-entropy effectively distills that best-of-N behavior back into the model's single-sample outputs. For verifiable domains like math or code, the 'reward' can simply be whether the final answer or unit test passes, removing the need for a learned reward model entirely.
Strategic Impact
Speed and scale
Language workflows can move faster without sacrificing consistency.
Access and reach
It expands access across languages and communication styles.
Clearer decisions
Teams can spend more time on judgment while automation handles repetition.
The Future of Rejection Sampling Fine-Tuning
RFT is central to modern post-training, often used before or alongside RL methods like PPO and DPO. Its appeal grows with cheap inference and strong automatic verifiers: as models get better at self-generating and self-checking, iterated rejection sampling supports synthetic-data and self-improvement loops. Expect tighter integration with reasoning models that produce verifiable chains of thought, and ongoing study of how to avoid reward hacking and diversity collapse when training repeatedly on a model's own outputs.
Real-World Implementation
Aligning Llama-style models by sampling multiple answers per prompt, keeping the highest reward-model scores, then SFT on those
Improving a math solver by generating many solutions and retaining only those that reach the correct, checkable answer
Code generation where candidates are kept only if they pass unit tests, then used as training data
Building synthetic instruction datasets by filtering a model's own best self-generated responses for the next training round
Risks & Guardrails
Hallucinated facts can quietly enter reports, support flows, or research outputs.
Prompt sensitivity can create inconsistent results across similar requests.
Sensitive text data may be exposed if access controls are weak.
Implementation Roadmap
Define output format, tone, and quality standards before rollout.
Ground responses with trusted sources whenever accuracy matters.
Keep a human review checkpoint for high-stakes outputs.
Track failure patterns and retrain prompts or workflows regularly.
Keep Exploring
Free newsletter
Keep up with AI in 3 minutes a day
One short email each weekday with the three AI stories that actually matter. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Rejection Sampling Fine-Tuning quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Next guide
QLoRA and 4-Bit Fine-Tuning
Frequently asked questions
What is Rejection Sampling Fine-Tuning?
Rejection Sampling Fine-Tuning (RFT) generates many candidate answers, keeps only the best-scoring ones, and retrains the model on those winners. It matters because it offers much of RLHF's benefit using straightforward supervised learning instead of complex reinforcement learning.
What is the core idea of Rejection Sampling Fine-Tuning?
RFT samples multiple responses, filters to the top-scoring ones, and fine-tunes the model on those winners.
In verifiable domains like math or code, what can serve as the reward?
For checkable tasks, RFT can use exact correctness or passing unit tests, avoiding a learned reward model.
Which model family famously used rejection sampling during alignment?
Meta described using rejection sampling as part of the post-training recipe for the Llama 2 and Llama 3 models.
Why might teams prefer RFT over policy-gradient RL like PPO?
RFT retrains with standard next-token loss on filtered samples, sidestepping the tuning difficulty and instability of policy-gradient methods.
Training on the maximum-reward samples effectively does what?
By learning from the top filtered responses, the model internalizes higher-quality behavior in a single generation.