Actor-Critic Methods Quiz

Answers and explanations appear as you take the quiz.

1. In an Actor-Critic method, what is the role of the 'critic'?
   • To estimate the value of states (or actions) and evaluate the actor's choices
   • To select which action to take
   • To store the replay buffer
   • To randomly explore the environment
2. What does the 'advantage' A(s,a) = Q(s,a) - V(s) measure?
   • The total episode reward
   • How much better an action is than the state's average value
   • The learning rate
   • The number of actions available
3. Why does adding a critic reduce variance compared to pure policy-gradient methods like REINFORCE?
   • It removes rewards from the equation
   • It makes the policy deterministic
   • The critic provides a baseline, so updates use a less noisy advantage signal
   • It increases the discount factor
4. Which capability do Actor-Critic methods have that plain value-based methods like Q-Learning handle awkwardly?
   • Discrete actions
   • Using rewards
   • Discounting the future
   • Continuous action spaces
5. Which of the following is an Actor-Critic-based algorithm widely used for RLHF on language models?
   • PPO (Proximal Policy Optimization)
   • K-means
   • Linear regression
   • Q-Learning
6. What signal does the actor typically use to update its policy?
   • The raw pixel input
   • The advantage or TD error provided by the critic
   • The replay buffer size
   • A fixed random number
7. An action has a positive advantage estimate. How does a policy-gradient actor typically use that signal?
   • Decrease its probability in every state
   • Set the critic’s value estimate to zero
   • Increase the probability of choosing that action in the state
   • Replace the policy with a Q-table
8. An action’s estimated advantage is negative. What direction should the actor update favor?
   • Increase it because the critic was surprised
   • Delete the state from the environment
   • Raise the reward discount factor automatically
   • Reduce the likelihood of that action relative to the policy baseline