技術指南

根據記錄的數據進行離策略評估

離策略評估使用舊策略記錄的資料來估計新決策策略的執行情況,而無需先部署新策略。

聆聽此指南AI 語音敘述 · ~3 min

本頁閱讀時間3分鐘
  1. 概述
  2. 深入探討
  3. 現實世界的實施
  4. The Future of Off-Policy Evaluation from Logged Data
  5. 付諸實踐
  6. 不斷探索
  7. 常見問題

概述

Inverse propensity scoring reweights outcomes by action probabilities, but reliable estimates require adequate policy overlap, correct propensities and careful handling of high variance and selective feedback.

深入探討

A policy maps context, such as a user and current page, to an action such as which item to display. Off-policy evaluation (OPE) estimates a target policy's expected outcomes using logs generated by a different behavior or logging policy. It can help screen a candidate before deployment, especially in recommendation, advertising and contextual bandits. Inverse propensity scoring (IPS) weights each logged reward by the ratio between the target policy's probability of choosing that action and the logging policy's probability. For deterministic policies, only records where the logged action matches the target contribute, scaled by the inverse logging propensity. This corrects selection under assumptions, but small logging propensities create large weights and high variance. Clipping weights can stabilize estimates while introducing bias. Self-normalized IPS divides by the sum of weights, and doubly robust methods combine propensity and reward models, each with distinct assumptions. Support or overlap is crucial. If the logging policy never chose an action in a context, the log contains no direct evidence of its outcome there. No weighting formula can recover that missing counterfactual without additional assumptions. Propensities must be logged accurately, and the reward outcome must be observed. Selective labels, delayed outcomes and interference can also undermine estimates. Use OPE as evidence, not a guarantee. Report effective sample size, weight distribution, overlap diagnostics and sensitivity to estimators. A large estimate driven by a few extreme weights should be treated cautiously. Candidate policy changes that move far outside historical support may require randomized exploration or a controlled online experiment. OPE complements offline predictive metrics but does not replace experimentation for every decision. Evaluation assumptions should be documented with the logging policy version, context features, action set and reward definition.

現實世界的實施

A recommender logged which item it showed, the context, click outcome and probability of selecting that item. An analyst uses those propensities to estimate a candidate policy's expected reward from the old log.

A target policy chooses actions that the logging policy almost never selected. The few matching outcomes receive large inverse weights, producing a noisy and potentially unstable estimate.

A team compares raw IPS with a self-normalized estimator and a doubly robust estimator, reporting assumptions and sensitivity rather than selecting whichever number is most favorable.

A policy evaluation lacks logged action probabilities. The analyst does not claim unbiased IPS results and instead collects better randomized logs or runs a controlled experiment.

The Future of Off-Policy Evaluation from Logged Data

OPE can help teams reduce risk by screening policies within the support of existing logs and identifying where evidence is too weak. Future data collection should record action propensities, exposure, context and outcomes with stable policy versions. Teams can combine OPE with small randomized experiments to improve support and validate estimates. Dashboards should show weight tails and effective sample size alongside policy value. When a candidate changes behavior substantially, the appropriate response may be to gather new evidence rather than extrapolate beyond the logs.

付諸實踐

為什麼這很重要

  • 成本與預算. 多年來,架構決策決定著效能和營運成本。
  • 更明確的決策. 技術教育幫助團隊選擇正確的堆疊,而不僅僅是最新的堆疊。
  • 品質管控. 更好的工程選擇可以減少生產中的可靠性事故。

留意

  • 優化一項基準測試可以隱藏更廣泛的系統弱點。
  • 基礎設施和維護成本常常被低估。
  • 隨著系統變得更加複雜,安全性和可觀察性差距可能會擴大。

後續步驟

  1. 在實施之前定義延遲、品質和成本目標。
  2. 在實際負載和資料條件下進行基準測試。
  3. 儀器監控錯誤、漂移和使用者影響。
  4. 在擴展之前準備回滾和事件回應路徑。

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

Take the Off-Policy Evaluation from Logged Data quiz

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

What is Off-Policy Evaluation from Logged Data?

Off-policy evaluation estimates how a new decision policy might perform using data logged by an older policy, without deploying the new policy first. Inverse propensity scoring reweights outcomes by action probabilities, but reliable estimates require adequate policy overlap, correct propensities and careful handling of high variance and selective feedback.

What does off-policy evaluation estimate?

OPE uses logged behavior and outcomes to estimate performance under a different policy.

Why does IPS divide by the logging propensity?

Inverse propensity weighting compensates for the logging policy's action selection under assumptions.

What happens when the logging policy assigns a very small probability to a chosen action?

Small propensities lead to large inverse weights, making estimates unstable.

What does lack of support mean for a target action?

If the logger never took an action in a context, OPE cannot directly learn its outcome there from those logs.

Why report effective sample size with weighted estimates?

Weight concentration can make a large raw dataset behave like a much smaller effective sample.