Grouped Reward Normalization in RLHF Quiz

Answers and explanations appear as you take the quiz.

1. In grouped reward normalization, what is each response's reward compared against?
   • The mean and standard deviation of its own prompt's group
   • A fixed global constant
   • The reward from the previous epoch
   • The model's parameter count
2. Which algorithm is most associated with grouped reward normalization?
   • Stochastic gradient descent
   • GRPO (Group Relative Policy Optimization)
   • K-means clustering
   • Backpropagation
3. What component of standard PPO does GRPO notably eliminate?
   • The reward model
   • The neural network entirely
   • The separate value network (critic)
   • The training data
4. What happens to a response whose reward is below its group's mean?
   • It receives positive advantage and is reinforced
   • It is deleted from the dataset
   • Its reward is doubled
   • It receives negative advantage and is discouraged
5. What does within-prompt reward normalization provide to a GRPO update?
   • A relative, scaled advantage for each response compared with others for the same prompt
   • A guarantee that all prompt difficulty differences disappear
   • A higher learning rate
   • A way to make every response receive the same reward
6. What happens when every response in a prompt group receives the same reward?
   • Every response gets a positive advantage
   • Centering gives no relative advantage; zero spread also requires numerical handling
   • The group mean doubles the reward
   • The model stops needing a policy objective
7. Which PPO-style update element does GRPO retain?
   • A separately learned value network
   • K-means clustering over tokens
   • A clipped policy objective
   • A language-model tokenizer
8. What can a KL penalty relative to a reference policy help constrain in GRPO?
   • The number of prompts in each group
   • The length of the tokenizer vocabulary
   • The resolution of generated images
   • How far the updated policy drifts from the reference policy