Iterative DPO and Online Preference Tuning Quiz

Answers and explanations appear as you take the quiz.

1. What does DPO avoid that traditional RLHF (PPO) requires?
   • Training a separate explicit reward model
   • Any training data
   • Using a neural network
   • Human preferences entirely
2. Why is iterative (online) DPO often better than running DPO once on a fixed dataset?
   • It needs no preference labels
   • It keeps training data on-policy and fresh as the model changes
   • It removes the reference model
   • It only works for images
3. What role does the reference model play in the DPO loss?
   • It generates the final user output
   • It labels the preferences
   • It anchors the policy via a log-probability ratio, controlling deviation
   • It replaces the tokenizer
4. In a single iteration of online DPO, what is the typical sequence?
   • Delete the model, then retrain from scratch
   • Only relabel old data without sampling
   • Train a CNN on the responses
   • Sample new responses, judge preferences, then run a DPO update
5. What does the beta hyperparameter in DPO control?
   • How strongly the policy deviates from the reference model
   • The vocabulary size
   • The number of GPUs
   • The image resolution
6. What is a key idea behind 'self-rewarding' iterative DPO?
   • The model uses no preferences ever
   • The model generates and judges its own response pairs to create training data
   • Humans must label every single pair
   • It only uses random rewards
7. Who can label response preferences in an online DPO iteration?
   • Only the tokenizer
   • Only the reference policy’s embedding layer
   • Human raters or an AI judge such as a reward model
   • A discriminator that never sees responses
8. What additional work does iterative DPO require compared with using a fixed preference dataset once?
   • Training no model parameters
   • Removing all evaluation from the pipeline
   • Storing every possible response in advance
   • Repeated response generation and preference labeling between updates