Guide-yakabatana mibvunzo · Zvakaoma Nhanho

Kugadziriswa Kwemubairo Mumapoka muRLHF Mibvunzo

Dzidza kuti GRPO inoenzanisa sei mhinduro dzesampled mukati meboka rinokurumidza, inobvisa anotsoropodza, uye inoshandisa chinangwa chePPO-maitiro ekugadzirisa matongerwo.

Nzira dzekutungamira dzinoenderanaYakakamurwa Reward Normalization MuRlhf
Mubvunzo 1 ye 8

Mumubairo wakaiswa muboka, mubairo wemhinduro yega yega unofananidzwa nei?

Tarisa mubvunzo wese8 mibvunzo

Mhinduro uye tsananguro zvinoonekwa paunenge uchitora bvunzo.

  1. Mumubairo wakaiswa muboka, mubairo wemhinduro yega yega unofananidzwa nei?
    • Kutsauswa kwepakati kweboka rayo rekukurumidza
    • A fixed global constant
    • Mubairo kubva panguva yapfuura
    • Iyo modhi parameter kuverenga
  2. Ndeipi algorithm inonyanya kuenderana neyakaiswa mubairo normalization?
    • Stochastic gradient descent
    • GRPO (Group Relative Policy Optimization)
    • K-zvinoreva kubatanidza
    • Kudzokera kumashure
  3. Ndeipi chikamu cheyakajairwa PPO icho GRPO inobvisa zvakanyanya?
    • Muenzaniso wemubayiro
    • Iyo neural network zvachose
    • Iyo yakasiyana kukosha network (critic)
    • Iyo data yekudzidziswa
  4. Chii chinoitika kumhinduro ine mubairo uri pasi pezvinorehwa neboka?
    • Inowana mukana wakanaka uye inosimbiswa
    • Inodzimwa kubva mudhata
    • Mubairo wayo wakapetwa kaviri
    • Inowana mukana wakashata uye inoodzwa mwoyo
  5. What does within-prompt reward normalization provide to a GRPO update?
    • A relative, scaled advantage for each response compared with others for the same prompt
    • A guarantee that all prompt difficulty differences disappear
    • A higher learning rate
    • A way to make every response receive the same reward
  6. What happens when every response in a prompt group receives the same reward?
    • Every response gets a positive advantage
    • Centering gives no relative advantage; zero spread also requires numerical handling
    • The group mean doubles the reward
    • The model stops needing a policy objective
  7. Which PPO-style update element does GRPO retain?
    • A separately learned value network
    • K-means clustering over tokens
    • A clipped policy objective
    • A language-model tokenizer
  8. What can a KL penalty relative to a reference policy help constrain in GRPO?
    • The number of prompts in each group
    • The length of the tokenizer vocabulary
    • The resolution of generated images
    • How far the updated policy drifts from the reference policy