Off-Policy Evaluation from Logged Data

A policy maps context, such as a user and current page, to an action such as which item to display. Off-policy evaluation (OPE) estimates a target policy's expected outcomes using logs generated by a different behavior or logging policy. It can help screen a candidate before deployment, especially in recommendation, advertising and contextual bandits. Inverse propensity scoring (IPS) weights each logged reward by the ratio between the target policy's probability of choosing that action and the logging policy's probability. For deterministic policies, only records where the logged action matches the target contribute, scaled by the inverse logging propensity. This corrects selection under assumptions, but small logging propensities create large weights and high variance. Clipping weights can stabilize estimates while introducing bias. Self-normalized IPS divides by the sum of weights, and doubly robust methods combine propensity and reward models, each with distinct assumptions. Support or overlap is crucial. If the logging policy never chose an action in a context, the log contains no direct evidence of its outcome there. No weighting formula can recover that missing counterfactual without additional assumptions. Propensities must be logged accurately, and the reward outcome must be observed. Selective labels, delayed outcomes and interference can also undermine estimates. Use OPE as evidence, not a guarantee. Report effective sample size, weight distribution, overlap diagnostics and sensitivity to estimators. A large estimate driven by a few extreme weights should be treated cautiously. Candidate policy changes that move far outside historical support may require randomized exploration or a controlled online experiment. OPE complements offline predictive metrics but does not replace experimentation for every decision. Evaluation assumptions should be documented with the logging policy version, context features, action set and reward definition.