Guide-linked quiz · Medium Level

Off-Policy Evaluation from Logged Data Quiz

Apply inverse propensity weighting and identify overlap, variance and logging requirements that limit counterfactual policy estimates.

Question 1 of 8

What does off-policy evaluation estimate?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. What does off-policy evaluation estimate?
    • A target policy's outcomes using data generated by another policy
    • The training accuracy of the logging model
    • A model's causal feature effects
    • The total number of logged requests
  2. Why does IPS divide by the logging propensity?
    • To make rewards nonnegative
    • To reweight observed actions according to how likely the old policy was to choose them
    • To eliminate the need for outcomes
    • To estimate the context distribution only
  3. What happens when the logging policy assigns a very small probability to a chosen action?
    • The candidate policy is unbiased automatically
    • The action outcome becomes known with certainty
    • The inverse weight can become large and increase variance
    • The reward is set to zero
  4. What does lack of support mean for a target action?
    • Every reward is missing
    • The action is always optimal
    • The target policy equals the logger
    • The logs contain little or no evidence for that action in the context
  5. Why report effective sample size with weighted estimates?
    • It changes the propensity values
    • It reveals whether a few large weights dominate the information
    • It measures the number of policy actions possible
    • It proves the reward model is correct
  6. What tradeoff can clipping IPS weights create?
    • Guaranteed unbiasedness
    • Higher sample size with no assumptions
    • Lower variance but potential bias
    • No effect on the estimate
  7. What must be logged to support ordinary IPS evaluation?
    • The candidate policy's future decisions
    • Only model predictions
    • Only final reward totals
    • Action probabilities and outcomes with relevant context
  8. When might a controlled experiment be needed instead of relying on OPE?
    • When the candidate policy acts outside historical support
    • Whenever a metric is positive
    • When propensities are accurately logged
    • When the target matches the logging policy

Keep testing yourself

More quizzes picked for your level.

View all quizzes