引導式測驗 · 硬 等級

RLHF 測驗中的分組獎勵標準化

了解 GRPO 如何比較提示組內的抽樣回應、刪除批評者以及使用 PPO 風格的目標進行策略更新。

問題 1 的 8

在分組獎勵標準化中,每個回應的獎勵與什麼進行比較?

預覽每個問題8 問題

當您參加測驗時,就會出現答案和解釋。

  1. 在分組獎勵標準化中,每個回應的獎勵與什麼進行比較?
    • 其自身提示組的平均值和標準差
    • 固定的全域常數
    • 上一個紀元的獎勵
    • 模型參數個數
  2. 哪種演算法與分組獎勵標準化最相關?
    • 隨機梯度下降
    • GRPO(群組相關策略優化)
    • K-均值聚類
    • 反向傳播
  3. GRPO 特別消除了標準 PPO 的哪些組成部分?
    • 獎勵模式
    • 完全神經網絡
    • 獨立的價值網絡(批評者)
    • 訓練資料
  4. 如果回應的獎勵低於其組別的平均值,會發生什麼情況?
    • 它獲得積極的優勢並得到加強
    • 它已從資料集中刪除
    • 其獎勵加倍
    • 它獲得負面優勢並被勸阻
  5. What does within-prompt reward normalization provide to a GRPO update?
    • A relative, scaled advantage for each response compared with others for the same prompt
    • A guarantee that all prompt difficulty differences disappear
    • A higher learning rate
    • A way to make every response receive the same reward
  6. What happens when every response in a prompt group receives the same reward?
    • Every response gets a positive advantage
    • Centering gives no relative advantage; zero spread also requires numerical handling
    • The group mean doubles the reward
    • The model stops needing a policy objective
  7. Which PPO-style update element does GRPO retain?
    • A separately learned value network
    • K-means clustering over tokens
    • A clipped policy objective
    • A language-model tokenizer
  8. What can a KL penalty relative to a reference policy help constrain in GRPO?
    • The number of prompts in each group
    • The length of the tokenizer vocabulary
    • The resolution of generated images
    • How far the updated policy drifts from the reference policy