Direct Preference Optimization Quiz

Answers and explanations appear as you take the quiz.

1. What does DPO eliminate compared to traditional RLHF?
   • The separate reward model and reinforcement learning loop
   • The need for any human preference data
   • The base pretrained language model
   • The use of log-probabilities
2. What data does a single DPO training example consist of?
   • A prompt and a numeric reward score
   • A prompt with a chosen and a rejected response
   • A prompt and a single correct answer
   • Only a reward model's output
3. What role does the reference model play in DPO?
   • It generates the rejected responses
   • It scores responses with explicit rewards
   • It is the frozen baseline the policy is measured against to limit drift
   • It replaces the training data
4. How does the DPO loss use a labeled preference pair?
   • It regresses both responses to the same scalar target
   • It trains a separate value critic to predict token-level returns
   • It maximizes the rejected response without a reference
   • It increases the chosen response’s policy-versus-reference log-ratio relative to the rejected response
5. What does the beta (temperature) parameter in DPO control?
   • How strongly the policy is kept close to the reference model
   • The learning rate of the optimizer
   • The number of preference pairs
   • The vocabulary size
6. Which alignment objective can learn from binary feedback that an output is desirable or undesirable?
   • PPO, which requires a value critic for every label
   • KTO, a prospect-theoretic objective for binary desirability feedback
   • DARTS, which searches differentiable network cells
   • Beam search, which changes model weights from rankings
7. In a standard DPO preference example, which response is preferred?
   • The response with more tokens by definition
   • The response with the lowest reference likelihood
   • The response selected by the preference labeler for the given prompt
   • A random response unrelated to the prompt
8. In DPO, what expression serves as the implicit reward for a response, up to an input-dependent constant?
   • A separately trained scalar reward-model head
   • A human-written point score for each token
   • The average sequence length of preference data
   • Beta times the policy/reference log-probability ratio