AI Glossary Term

What does “DPO (Direct Preference Optimization)” mean?

Definition

A training method that fine-tunes models directly on preference pairs without needing a separate reward model.

Browse the full AI GlossaryExplore all free AI guides

How DPO (Direct Preference Optimization) works

Diagram for DPO (Direct Preference Optimization) comparing RLHF (Train reward model, Run reinforcement learning) versus DPO (Train directly on preferred vs rejected answers, One simpler stage).RLHFTrain reward modelRun reinforcementlearningDPOTrain directly onpreferred vsrejected answersOne simpler stagevs

Example

Given pairs of replies where people preferred one, DPO adjusts the model to make the preferred reply more likely than the rejected one, without training a separate reward model.

Why it matters

It simplifies preference training and is more stable and cheaper than full reinforcement-learning pipelines.

Common questions

Who introduced DPO?

A 2023 paper by researchers at Stanford.

Does it match RLHF in quality?

Often comparable on many tasks, though results vary with data and setup.

What data does it need?

Pairs of responses to the same prompt labeled preferred and rejected.

Related terms

Reinforcement Learning from Human Feedback (RLHF)A training method that uses human preference signals to shape model behavior.Post-trainingTraining steps applied after pretraining, such as instruction tuning, preference optimization, and safety tuning.Continual LearningTraining approaches that let a model keep learning from new data without forgetting prior knowledge.Fine-TuningContinuing training on domain-specific data to adapt a pre-trained model to a specific task.Gradient DescentAn optimization method that updates parameters in the direction that reduces error.Instruction TuningFine-tuning a model on instruction-response pairs to improve task following.

Learn more in our free guides

AI Inference OptimizationBidirectional Recurrent NetworksSecond-Order Optimization and Newton MethodsOdds Ratio Preference Optimization

Test yourself

AI Inference Optimization QuizBidirectional Recurrent Networks QuizSecond-Order Optimization and Newton Methods Quiz

See also

Tree of ThoughtsQLoRAReActFlash AttentionAgentic LoopMulti-Head AttentionMCP (Model Context Protocol)Positional Encoding