引導式測驗 · 中等 等級
直接偏好優化測驗
查看 DPO 偏好對、參考策略、隱性獎勵、貝塔值、變體和直接偏好訓練的限制。
問題 1 的 8
與傳統的 RLHF 相比,DPO 消除了什麼?
預覽每個問題8 問題
當您參加測驗時,就會出現答案和解釋。
- 與傳統的 RLHF 相比,DPO 消除了什麼?
- 單獨的獎勵模型與強化學習循環
- 需要任何人類偏好數據
- 基礎預訓練語言模型
- 對數機率的使用
- 單一 DPO 訓練範例包含哪些資料?
- 提示和數字獎勵分數
- 帶有選擇和拒絕回應的提示
- 一個提示和一個正確答案
- 僅獎勵模型的輸出
- 參考模型在 DPO 中發揮什麼作用?
- 它產生拒絕的回應
- 它透過明確的獎勵對回應進行評分
- 這是衡量政策以限制漂移的凍結基線
- 它取代了訓練數據
- DPO損失如何使用標記的偏好對?
- It regresses both responses to the same scalar target
- It trains a separate value critic to predict token-level returns
- It maximizes the rejected response without a reference
- It increases the chosen response’s policy-versus-reference log-ratio relative to the rejected response
- DPO 中的 beta(溫度)參數控制什麼?
- 策略與參考模型的接近程度
- 優化器的學習率
- 偏好對的數量
- 詞彙量大小
- Which alignment objective can learn from binary feedback that an output is desirable or undesirable?
- PPO, which requires a value critic for every label
- KTO, a prospect-theoretic objective for binary desirability feedback
- DARTS, which searches differentiable network cells
- Beam search, which changes model weights from rankings
- In a standard DPO preference example, which response is preferred?
- The response with more tokens by definition
- The response with the lowest reference likelihood
- The response selected by the preference labeler for the given prompt
- A random response unrelated to the prompt
- In DPO, what expression serves as the implicit reward for a response, up to an input-dependent constant?
- A separately trained scalar reward-model head
- A human-written point score for each token
- The average sequence length of preference data
- Beta times the policy/reference log-probability ratio