Q-Learning Quiz

Answers and explanations appear as you take the quiz.

1. What does the Q-value Q(s, a) represent?
   • The expected long-term reward of taking action a in state s, then acting optimally
   • The probability of being in state s
   • The number of times action a has been tried
   • The immediate reward only for action a
2. Why is Q-Learning described as 'off-policy'?
   • It requires a model of the environment
   • It can learn the optimal policy while following a different, exploratory policy
   • It only works offline on stored data
   • It ignores rewards entirely
3. In the update rule, what does the discount factor gamma control?
   • The exploration rate
   • The size of the state table
   • How much future rewards are weighted relative to immediate ones
   • The number of episodes
4. What is the temporal-difference (TD) error in Q-Learning?
   • The difference between two states
   • The total reward of an episode
   • The learning rate times gamma
   • r + gamma*max Q(s',a') minus the current Q(s,a) estimate
5. Why does plain tabular Q-Learning struggle with large problems like video games from pixels?
   • The number of distinct states is too large to store in a table
   • It cannot use rewards
   • It only works with two actions
   • Gamma must equal 1
6. What is the purpose of epsilon-greedy action selection?
   • To always pick the highest Q-value action
   • To balance exploring new actions with exploiting known good ones
   • To set the discount factor
   • To store the Q-table
7. What exploration condition appears in the tabular Q-learning convergence result?
   • Only the greedy action is ever sampled
   • Each state must be visited exactly once
   • Every state-action pair must be visited repeatedly under suitable learning conditions
   • The transition model must be known in advance
8. Why is Q-learning called model-free?
   • It never observes the environment
   • It does not use rewards
   • It requires a complete transition table before learning
   • It updates values from sampled rewards and next states without storing transition probabilities