Language AI GUIDE
Rejection Sampling Fine-Tuning
Rejection-sampling fine-tuning generates candidate responses, filters them with a reward model or verifier, and uses selected examples for supervised training.
Listen to this guideAI-narrated · ~2 min
On this page2 min read
Overview
It can be simpler than policy-gradient optimization, but quality depends on the candidate pool and scorer.
Deep Dive
A rejection-sampling fine-tuning loop prompts a policy to generate multiple candidates, scores them with a reward model or task-specific verifier, and retains examples meeting a selection rule. The model is then fine-tuned with supervised learning on selected responses. In verifiable tasks, tests or exact-answer checks can provide a reward; open-ended tasks often rely on learned preference scores. Llama 2-Chat’s paper describes rejection sampling as one stage of its alignment pipeline. RFT can transfer best-of-N behavior into a model, but cannot select answers absent from the candidate pool and may overfit or exploit an imperfect scorer. It is not equivalent to every RLHF recipe and does not guarantee higher quality or diversity.
Real-World Implementation
Aligning Llama-style models by sampling multiple answers per prompt, keeping the highest reward-model scores, then SFT on those
Improving a math solver by generating many solutions and retaining only those that reach the correct, checkable answer
Code generation where candidates are kept only if they pass unit tests, then used as training data
Building synthetic instruction datasets by filtering a model's own best self-generated responses for the next training round
The Future of Rejection Sampling Fine-Tuning
RFT is central to modern post-training, often used before or alongside RL methods like PPO and DPO. Its appeal grows with cheap inference and strong automatic verifiers: as models get better at self-generating and self-checking, iterated rejection sampling supports synthetic-data and self-improvement loops. Expect tighter integration with reasoning models that produce verifiable chains of thought, and ongoing study of how to avoid reward hacking and diversity collapse when training repeatedly on a model's own outputs.
Putting it into practice
Why it matters
- Speed and scale. Language workflows can move faster without sacrificing consistency.
- Access and reach. It expands access across languages and communication styles.
- Clearer decisions. Teams can spend more time on judgment while automation handles repetition.
Watch out for
- Hallucinated facts can quietly enter reports, support flows, or research outputs.
- Prompt sensitivity can create inconsistent results across similar requests.
- Sensitive text data may be exposed if access controls are weak.
Next steps
- Define output format, tone, and quality standards before rollout.
- Ground responses with trusted sources whenever accuracy matters.
- Keep a human review checkpoint for high-stakes outputs.
- Track failure patterns and retrain prompts or workflows regularly.
Keep Exploring
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Rejection Sampling Fine-Tuning quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Frequently asked questions
What is Rejection Sampling Fine-Tuning?
Rejection-sampling fine-tuning generates candidate responses, filters them with a reward model or verifier, and uses selected examples for supervised training. It can be simpler than policy-gradient optimization, but quality depends on the candidate pool and scorer.
Which loop best describes how Rejection Sampling Fine-Tuning improves a model?
RFT samples multiple responses, filters to the top-scoring ones, and fine-tunes the model on those winners.
In verifiable domains like math or code, what can serve as the reward?
For checkable tasks, RFT can use exact correctness or passing unit tests, avoiding a learned reward model.
Which model family does the cited paper describe using rejection sampling in its alignment pipeline?
The Llama 2 paper describes rejection sampling as one stage of the Llama 2-Chat alignment pipeline.
Why might a team choose rejection-sampling fine-tuning over a policy-gradient approach for some tasks?
RFT uses generated and filtered examples for supervised updates, though generation, scoring, and data quality still incur costs.
Training on the maximum-reward samples effectively does what?
By learning from the top filtered responses, the model internalizes higher-quality behavior in a single generation.
Keep learning
Related guides
More guides picked for this topic