Maswali yanayounganishwa na mwongozo · Ngumu Kiwango

Urekebishaji wa Zawadi Uliopangwa katika Makundi katika Maswali ya RLHF

Jifunze jinsi GRPO inavyolinganisha majibu yaliyotolewa katika kikundi cha papo hapo, kuondoa mkosoaji na kutumia lengo la mtindo wa PPO kwa masasisho ya sera.

Njia za mwongozo zinazohusianaUrekebishaji wa Zawadi Uliopangwa Katika Makundi Katika Rlhf
Swali 1 ya 8

Katika urekebishaji wa zawadi uliowekwa katika vikundi, je, kila jibu lina thawabu gani ikilinganishwa na?

Hakiki kila swali8 maswali

Majibu na maelezo huonekana unapojibu maswali.

  1. Katika urekebishaji wa zawadi uliowekwa katika vikundi, je, kila jibu lina thawabu gani ikilinganishwa na?
    • Mkengeuko wa wastani na wa kawaida wa kikundi chake cha kidokezo
    • A fasta kimataifa
    • Zawadi kutoka enzi iliyopita
    • Hesabu ya parameta ya mfano
  2. Ni algorithm gani inayohusishwa zaidi na urekebishaji wa zawadi uliowekwa katika vikundi?
    • Kushuka kwa gradient ya Stochastic
    • GRPO (Uboreshaji wa Sera Husika ya Kikundi)
    • K-inamaanisha kuunganisha
    • Uenezaji wa nyuma
  3. Ni sehemu gani ya PPO ya kawaida ambayo GRPO inaondoa haswa?
    • Mfano wa malipo
    • Mtandao wa neva kabisa
    • Mtandao wa thamani tofauti (mkosoaji)
    • Takwimu za mafunzo
  4. Nini kinatokea kwa jibu ambalo malipo yake yako chini ya wastani wa kundi lake?
    • Inapokea faida nzuri na inaimarishwa
    • Inafutwa kutoka kwa mkusanyiko wa data
    • Malipo yake ni maradufu
    • Inapokea faida hasi na inakatishwa tamaa
  5. What does within-prompt reward normalization provide to a GRPO update?
    • A relative, scaled advantage for each response compared with others for the same prompt
    • A guarantee that all prompt difficulty differences disappear
    • A higher learning rate
    • A way to make every response receive the same reward
  6. What happens when every response in a prompt group receives the same reward?
    • Every response gets a positive advantage
    • Centering gives no relative advantage; zero spread also requires numerical handling
    • The group mean doubles the reward
    • The model stops needing a policy objective
  7. Which PPO-style update element does GRPO retain?
    • A separately learned value network
    • K-means clustering over tokens
    • A clipped policy objective
    • A language-model tokenizer
  8. What can a KL penalty relative to a reference policy help constrain in GRPO?
    • The number of prompts in each group
    • The length of the tokenizer vocabulary
    • The resolution of generated images
    • How far the updated policy drifts from the reference policy