Length Normalization in Preference Optimization Quiz

Answers and explanations appear as you take the quiz.

1. What undesirable behavior does length normalization in DPO primarily aim to prevent?
   • Models favoring longer responses regardless of actual quality
   • Models refusing to answer questions
   • Models hallucinating facts
   • Models responding too slowly
2. Why does standard DPO's implicit reward tend to increase with response length?
   • It multiplies the reward by temperature
   • It is a sum of per-token log-probability differences
   • It uses a fixed bonus per sentence
   • It penalizes short answers explicitly
3. How does SimPO address the length bias?
   • It doubles the reference model's weight
   • It truncates all responses to a fixed length
   • It uses the average per-token log-probability as the reward
   • It removes the preference data entirely
4. A notable structural difference between SimPO and standard DPO is that SimPO:
   • Requires twice as much training data
   • Only works on images
   • Disables gradient descent
   • Does not use a separate reference model
5. Which evaluation practice helps confirm a model improved in quality rather than just length?
   • Reporting a length-controlled win rate
   • Counting total parameters
   • Measuring GPU temperature
   • Increasing the batch size
6. Which chat-based benchmarks did the SimPO paper use in its reported evaluations?
   • ImageNet, COCO, and GLUE
   • AlpacaEval 2, MT-Bench, and Arena-Hard
   • LibriSpeech, WER, and BLEU
   • MMLU, GSM8K, and HumanEval only
7. What does SimPO’s target reward margin encourage between a preferred and rejected response?
   • Identical scores for both responses
   • A longer preferred response regardless of quality
   • A larger score difference in favor of the preferred response
   • A new reference model for every token
8. What risk can overly aggressive length normalization create?
   • It guarantees all answers have the same factual accuracy
   • It makes the reference model’s vocabulary larger
   • It prevents evaluators from comparing responses
   • It can favor answers that are too terse to include necessary detail