Offline Reinforcement Learning Quiz

Answers and explanations appear as you take the quiz.

1. What defines offline (batch) reinforcement learning?
   • Learning from a fixed dataset without new environment interaction
   • Learning only through continuous live exploration
   • Training without any reward signal
   • Running inference without any model
2. What is the central technical challenge in offline RL?
   • Lack of any data
   • Distributional shift and overestimation of out-of-distribution actions
   • Excessive online exploration
   • Too few parameters in the network
3. What does D4RL provide to researchers evaluating offline reinforcement-learning algorithms?
   • A guarantee that an offline policy is safe to deploy
   • A new optimizer that removes the need for logged data
   • A benchmark suite of datasets and tasks for offline RL evaluation
   • A dataset containing only online interactions collected by each test policy
4. How does Conservative Q-Learning (CQL) combat overestimation?
   • By inflating values for unseen actions
   • By removing the reward entirely
   • By exploring more in the real world
   • By regularizing Q-values down for out-of-distribution actions
5. How does the Decision Transformer reframe offline RL?
   • As sequence modeling that predicts actions conditioned on desired return
   • As clustering of states
   • As pure random search
   • As supervised image classification
6. Why can't standard online RL algorithms simply be used as-is offline?
   • They have no neural network
   • They cannot collect corrective data to fix value overestimation
   • They never use rewards
   • They require no policy
7. How does BCQ reduce reliance on actions absent from an offline dataset?
   • It adds random actions to every training transition
   • It assumes every action has been observed equally often
   • It constrains candidate actions to stay near actions the dataset supports
   • It removes the action-value function and reward
8. What value-learning method is central to the cited Implicit Q-Learning paper?
   • Maximum-likelihood fitting of a language tokenizer
   • Uniform regression over actions sampled from a live environment
   • A supervised image-classification loss
   • Expectile regression over values associated with dataset actions