引导式测验 · 中等 等级

直接偏好优化测验

查看 DPO 偏好对、参考策略、隐性奖励、贝塔值、变体和直接偏好训练的限制。

相关引导路径直接偏好优化
问题 1 的 8

与传统的 RLHF 相比，DPO 消除了什么？

预览每个问题8 问题

当您参加测验时，就会出现答案和解释。

  1. 与传统的 RLHF 相比，DPO 消除了什么？
    • 单独的奖励模型和强化学习循环
    • 需要任何人类偏好数据
    • 基础预训练语言模型
    • 对数概率的使用
  2. 单个 DPO 训练示例包含哪些数据？
    • 提示和数字奖励分数
    • 带有选择和拒绝响应的提示
    • 一个提示和一个正确答案
    • 仅奖励模型的输出
  3. 参考模型在 DPO 中发挥什么作用？
    • 它生成拒绝的响应
    • 它通过明确的奖励对响应进行评分
    • 这是衡量政策以限制漂移的冻结基线
    • 它取代了训练数据
  4. How does the DPO loss use a labeled preference pair?
    • It regresses both responses to the same scalar target
    • It trains a separate value critic to predict token-level returns
    • It maximizes the rejected response without a reference
    • It increases the chosen response’s policy-versus-reference log-ratio relative to the rejected response
  5. DPO 中的 beta（温度）参数控制什么？
    • 策略与参考模型的接近程度
    • 优化器的学习率
    • 偏好对的数量
    • 词汇量大小
  6. Which alignment objective can learn from binary feedback that an output is desirable or undesirable?
    • PPO, which requires a value critic for every label
    • KTO, a prospect-theoretic objective for binary desirability feedback
    • DARTS, which searches differentiable network cells
    • Beam search, which changes model weights from rankings
  7. In a standard DPO preference example, which response is preferred?
    • The response with more tokens by definition
    • The response with the lowest reference likelihood
    • The response selected by the preference labeler for the given prompt
    • A random response unrelated to the prompt
  8. In DPO, what expression serves as the implicit reward for a response, up to an input-dependent constant?
    • A separately trained scalar reward-model head
    • A human-written point score for each token
    • The average sequence length of preference data
    • Beta times the policy/reference log-probability ratio