AI Alignment Quiz

Answers and explanations appear as you take the quiz.

1. In alignment research, what does the term "reward hacking" describe?
   • A model refusing to optimize its reward because the task seems harmful
   • An agent exploiting flaws in its reward signal instead of doing the intended task
   • An attacker stealing the reward model's weights to retrain a competing system
   • Developers manually raising reward values to speed up convergence during training
2. In RLHF, what is the role of the reward model?
   • It replaces human raters entirely by writing the reference answers for fine-tuning
   • It stores a fixed list of banned words that the policy is never allowed to output
   • It generates new pretraining text so the base model has more data to learn from
   • It learns from human preference comparisons and scores outputs to guide the policy
3. What does goal misspecification mean in the guide?
   • A model behaves differently when it recognizes a test setting
   • The written objective fails to represent what people actually intend
   • An agent finds a shortcut in a proxy metric while optimizing it
   • A model refuses every task after a safety update
4. Which description matches instrumental convergence in the guide?
   • A user changes their prompt after a refusal
   • A system may seek power or resources because they help achieve many possible goals
   • A test set is held out from training
   • A reward model stores written principles
5. Which modern alignment approach is named in the guide?
   • Credit scoring notices
   • Post-market incident reporting
   • Dataset deduplication
   • Constitutional AI
6. What does "scalable oversight" research try to solve?
   • How to reduce the number of parameters while keeping accuracy constant
   • How humans can supervise AI on tasks too hard for them to evaluate directly
   • How to serve a model to millions of users at once without increasing latency
   • How to scale training clusters across multiple data centers efficiently
7. Why does preference optimization not prove that a model has adopted human intent as an internal goal?
   • It improves average responses but does not prove stable behavior under shift or pressure
   • It prevents models from using human preference data
   • It guarantees a model will ignore every written instruction
   • It measures only GPU efficiency
8. Why might an alignment evaluation include tests for evaluation awareness?
   • To measure the number of human raters in training
   • To determine whether a reward model has enough parameters
   • To see whether a model changes behavior when it recognizes a test setting
   • To prove that a system is conscious