Reinforcement Learning From Human Feedback Quiz

Answers and explanations appear as you take the quiz.

1. What is the main purpose of RLHF for a language model?
   • To align its outputs with human preferences for helpfulness and safety
   • To increase the model's parameter count
   • To compress the model for faster inference
   • To translate the model into other languages
2. What does the reward model in RLHF actually learn to do?
   • Generate the final responses to users
   • Translate prompts into a reward currency
   • Assign a scalar score reflecting how much humans would prefer a given response
   • Decide the model's learning rate
3. What does a KL-divergence penalty against a reference policy aim to regularize in an RLHF objective?
   • Every factual error with a complete correctness guarantee
   • How far the optimized policy departs from the reference
   • The tokenizer vocabulary’s spelling
   • The number of human annotators automatically
4. How are preference data typically collected for the reward model?
   • Humans write brand-new answers from scratch
   • The model rates its own confidence
   • Random labels are assigned automatically
   • Humans compare two model responses to the same prompt and choose the better one
5. What advantage does DPO (Direct Preference Optimization) offer over the classic RLHF pipeline?
   • It requires no data at all
   • It guarantees the model can never be wrong
   • It optimizes the policy directly from preference pairs, skipping a separate reward model and PPO loop
   • It eliminates the need for pretraining
6. In RLAIF and Constitutional AI, where does much of the feedback come from?
   • From an AI model generating or critiquing responses, reducing human labeling
   • Only from paid human annotators
   • From random number generators
   • From the model's pretraining corpus directly
7. An RLHF response receives a high learned reward but contains a false claim. What does this show?
   • High reward proves the claim is true
   • The response must have been generated without tokens
   • The reference model’s vocabulary disappeared
   • A preference-based reward is a proxy and can be mistaken
8. Why should an RLHF team examine whose preference comparisons were collected?
   • Annotator identity determines the GPU clock rate
   • The judgments and task distribution shape what the model is rewarded for
   • Every population has identical response preferences
   • Preference data affect only file compression