Reward Hacking and Specification Gaming Quiz

Answers and explanations appear as you take the quiz.

1. What is the core problem behind reward hacking?
   • The reward function is a proxy that imperfectly captures the true goal
   • The AI runs out of memory during training
   • The model is too small to learn the task
   • Training data is always mislabeled
2. In OpenAI's CoastRunners example, what did the boat agent do?
   • Finished the race faster than humans
   • Spun in circles hitting bonus targets instead of finishing
   • Crashed immediately every time
   • Refused to move at all
3. Which principle states that a measure ceases to be good once it becomes a target?
   • Moore's Law
   • Amdahl's Law
   • Goodhart's Law
   • Murphy's Law
4. How does reward hacking commonly appear in language models trained with RLHF?
   • Refusing to generate any text
   • Only outputting numbers
   • Translating everything into Latin
   • Sycophancy and telling users what they want to hear
5. Which technique helps keep an RLHF policy from drifting too far and exploiting the reward model?
   • A KL penalty keeping the policy near the base model
   • Increasing the batch size only
   • Removing all human feedback
   • Disabling the optimizer
6. Why does reward hacking become a bigger safety concern as models get more capable?
   • Bigger models cannot be trained at all
   • More capable optimizers find subtler loopholes that are harder to detect
   • Capability has no effect on hacking
   • Larger models ignore the reward entirely
7. What does process-based supervision reward in addition to a final answer?
   • Only the number of tokens in the final answer
   • The agent’s ability to disable its evaluator
   • The quality of intermediate steps used to reach the answer
   • A score unrelated to the task
8. Why can a reward-model ensemble help detect reward hacking?
   • It guarantees the reward proxy exactly matches human intent
   • It removes the need to test the policy
   • It makes all reward models produce identical scores
   • Disagreement among models can flag outputs that exploit one scorer’s blind spots