Group Relative Policy Optimization Quiz

Answers and explanations appear as you take the quiz.

1. Which separately learned estimator does the original GRPO approach avoid training?
   • The policy that generates completions
   • A value or critic model
   • Every possible reward function
   • The reference policy’s vocabulary
2. How does GRPO compute the advantage for a given completion?
   • By querying a learned value head at each token
   • By using the raw reward with no normalization
   • By comparing it to completions from a different prompt
   • By standardizing its reward against the mean and standard deviation of its group of sibling completions
3. Which models made GRPO well known?
   • DeepSeek's DeepSeekMath and DeepSeek-R1 reasoning models
   • OpenAI's GPT-2 series
   • Google's original BERT
   • Meta's Llama 2 base models
4. What role does the KL-divergence term play in GRPO?
   • It generates the group of completions
   • It replaces the reward model
   • It keeps the trained policy from drifting too far from a frozen reference model
   • It computes the per-token value estimate
5. Why is GRPO especially attractive for training reasoning models on math or code?
   • It requires no reward signal at all
   • It only works with human preference labels
   • It cannot use sampled completions
   • It works naturally with verifiable, rule-based rewards (e.g., did the answer match or did tests pass) scored across a group of sampled solutions
6. What is one known bias that follow-up work (like Dr. GRPO) tries to correct in vanilla GRPO?
   • A tendency to ignore the reward entirely
   • Length and difficulty biases introduced by how rewards are normalized
   • An inability to sample more than one completion
   • A requirement for an enormous critic network
7. Three GRPO completions for one prompt receive rewards 2, 4, and 6. Which completion has zero centered advantage?
   • The completion rewarded 2
   • The completion rewarded 6
   • The completion rewarded 4
   • All completions because rewards differ
8. Why compare GRPO reward statistics among completions of the same prompt?
   • They provide a relative baseline for alternative answers to that prompt
   • They require unrelated prompts to have equal difficulty
   • They replace checking whether an answer is correct
   • They make the highest reward identical across all tasks