Reward Modeling Quiz

Answers and explanations appear as you take the quiz.

1. What is the primary output of a reward model for a given prompt-response pair?
   • A single scalar score
   • A full text response
   • A probability distribution over the vocabulary
   • A list of preferred tokens
2. What feedback format did the InstructGPT paper collect to train its reward model?
   • A guaranteed correctness label for every possible answer
   • Human rankings of several model responses, converted into pairwise preferences
   • A single sentiment score for each training prompt
   • Only binary safety labels with no response comparisons
3. What does the standard reward-model loss optimize?
   • The absolute reward value of every response
   • The next-token prediction accuracy
   • That the chosen response scores higher than the rejected one
   • The length of responses
4. What is 'reward hacking'?
   • Stealing a competitor's reward model
   • A method for speeding up training
   • Encrypting the reward signal
   • The policy exploiting flaws in the reward model to get high scores without genuine quality
5. How is a reward model architecturally derived from a language model?
   • By replacing the token-prediction head with a single scalar output head
   • By adding more attention layers
   • By removing all the transformer layers
   • By doubling the vocabulary
6. How is a reward model used at inference time in best-of-N sampling?
   • Training N reward models in parallel
   • Generating multiple responses and using the reward model to pick the highest-scoring one
   • Selecting the N most common tokens
   • Averaging N model checkpoints
7. Why might an RLHF study use an ensemble of reward models during training?
   • To guarantee that reward hacking is impossible
   • To replace the human preference data with exact correctness labels
   • To estimate disagreement or uncertainty across learned preference scorers
   • To ensure every candidate receives the same score
8. Why should reward-model evaluations include held-out comparisons?
   • To ensure training comparisons are never used
   • To make every output identical
   • To guarantee the reward is unbiased
   • To test whether preferences generalize beyond the training comparisons