Proximal Policy Optimization Quiz

Answers and explanations appear as you take the quiz.

1. What problem does PPO's 'clipping' primarily address?
   • Overly large policy updates that destabilize training
   • Slow data loading
   • Vocabulary size limits
   • Memory usage during inference
2. In language-model RLHF with PPO, where does the reward signal usually come from?
   • Direct human scoring of every token
   • A trained reward model
   • The optimizer's learning rate
   • The size of the vocabulary
3. What does the KL-divergence penalty do in PPO-based RLHF?
   • Speeds up token generation
   • Increases the reward model's accuracy
   • Keeps the fine-tuned model from drifting too far from the reference model
   • Removes the need for a value network
4. What is the role of the value (critic) network in PPO?
   • To generate the final text output
   • To store the training data
   • To replace the reward model entirely
   • To estimate expected return so advantages can be computed
5. What does the 'advantage' represent in PPO?
   • How much better an action was than the expected baseline
   • The model's total parameter count
   • The clipping range
   • The KL penalty coefficient
6. What distinguishes GRPO from PPO in the DeepSeekMath paper?
   • It removes all policy sampling and optimizes only supervised labels
   • It estimates relative advantages from groups of sampled responses without a separate value critic
   • It uses a trained reward model as a value network
   • It guarantees zero policy drift after one update
7. What probability ratio does PPO clip in its surrogate objective?
   • The reward divided by the number of prompts
   • The value estimate divided by the KL coefficient
   • The new policy’s action probability divided by the old policy’s action probability
   • The sequence length divided by batch size
8. Why is PPO called an on-policy method in its standard form?
   • It trains only on a static image dataset
   • It never samples actions
   • It updates only a reward model
   • It collects trajectories using the current policy being optimized