Ikibazo gihuza ibibazo · Hagati Urwego

Ikibazo cya Politiki Yegeranye

Ongera usubiremo PPO, ibipimo bya politiki, inkomoko y'ibihembo, ibihano bya KL, abanegura, ibyiza, hamwe nururimi-rwerekana imipaka RLHF. Reba ibitekerezo n'ibipimo.

Inzira zijyanye no kuyoboraGukwirakwiza Politiki Yegeranye
Ikibazo 1 Bya 8

Ni ikihe kibazo 'gukuramo' PPO gikemura cyane cyane?

Reba buri kibazo8 ibibazo

Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.

  1. Ni ikihe kibazo 'gukuramo' PPO gikemura cyane cyane?
    • Kurenza cyane politiki ivugurura ihagarika amahugurwa
    • Buhoro buhoro gupakira amakuru
    • Ingano yamagambo ntarengwa
    • Gukoresha kwibuka mugihe cyo gufata umwanzuro
  2. Mundimi-moderi RLHF hamwe na PPO, ibimenyetso byigihembo mubisanzwe bituruka he?
    • Gutanga amanota yumuntu kuri buri kimenyetso
    • Icyitegererezo cyo guhemba
    • Igipimo cyo kwiga neza
    • Ingano yamagambo
  3. Niki gihano KL-divergence ikora muri RLHF ishingiye kuri PPO?
    • Kwihutisha ibisekuruza
    • Yongera ibihembo byicyitegererezo
    • Komeza icyitegererezo cyiza cyo gutembera kure yicyitegererezo
    • Kuraho ibikenewe byurusobe rwagaciro
  4. Ni uruhe ruhare rw'urusobe rw'agaciro (kunegura) muri PPO?
    • Kubyara inyandiko yanyuma
    • Kubika amakuru yamahugurwa
    • Gusimbuza icyitegererezo cyigihembo
    • Kugereranya ibyateganijwe kugaruka bityo inyungu zirashobora kubarwa
  5. Niki 'akarusho' kagereranya muri PPO?
    • Nigute ibikorwa byari byiza kuruta ibyateganijwe
    • Icyitegererezo cyuzuye cyo kubara
    • Urwego rwo gukuramo
    • Coefficient ya KL
  6. What distinguishes GRPO from PPO in the DeepSeekMath paper?
    • It removes all policy sampling and optimizes only supervised labels
    • It estimates relative advantages from groups of sampled responses without a separate value critic
    • It uses a trained reward model as a value network
    • It guarantees zero policy drift after one update
  7. What probability ratio does PPO clip in its surrogate objective?
    • The reward divided by the number of prompts
    • The value estimate divided by the KL coefficient
    • The new policy’s action probability divided by the old policy’s action probability
    • The sequence length divided by batch size
  8. Why is PPO called an on-policy method in its standard form?
    • It trains only on a static image dataset
    • It never samples actions
    • It updates only a reward model
    • It collects trajectories using the current policy being optimized