InoteveraGaidhi rinotevera
Proximal Policy Optimization
Mutauro AI
Nhungamiro yehunyanzvi
Boka Relative Policy Optimization
Boka Relative Policy Optimization (GRPO) inzira yekusimbisa-yekudzidza yemhando dzemitauro inotonga mhinduro yega yega vachipokana neboka rehama mhinduro kune imwecheteyo kukurumidza, kubvisa yakaparadzana kukosha network inoshandiswa nePPO.
Teerera kugwaro iriAI-yakarondedzerwa · ~2 min
Pa peji ino2 min verenga
Pfupiso
Yakava nemukurumbira seyakanyanya kudzidzisa trick kuseri kweDeepSeek's mafungiro emhando.
Kudzika Kwakadzika
GRPO is a variant of policy-gradient reinforcement learning designed to make RL fine-tuning of large language models cheaper and more stable. Many PPO implementations train a value model to estimate returns; its size and cost depend on the implementation. GRPO removes that critic entirely. For each prompt it samples a group of completions (say 8-64), scores them all with a reward signal, and then computes each completion's advantage by standardizing its reward against the group's mean and standard deviation. Above-average answers are reinforced and below-average ones suppressed. A KL-divergence term keeps the model close to a reference policy. Introduced by DeepSeek, it powered DeepSeekMath and the DeepSeek-R1 reasoning models.
Kuitwa muNyika yeChokwadi
Kudzidzisa DeepSeek-R1 uye DeepSeekMath kuburitsa refu ketani-ye-yekufunga kufunga uchishandisa mutemo-wakavakirwa kurongeka mibairo pamatambudziko esvomhu.
Yakanaka-tuning kodhi-chizvarwa modhi apo yega yega mhinduro inopihwa kana ichipfuura bvunzo dzeyuniti, uye boka rinojairwa kusarudza vanokunda.
Open-source RLHF mapaipi (semuenzaniso, muTRL uye verl maraibhurari) uchishandisa GRPO kurongedza mhando dzekutaura pasina kubhadhara kune yakaparadzana kukosha network.
Kuvandudza kuraira-kutevera kana kuchengetedza maitiro nekuenzanisa akati wandei mhinduro nekukasira uye kupa mubairo iwo mubairo wemubairo mitengo yepamusoro kune vezera ravo.
Ramangwana reBoka Relative Policy Optimization
GRPO yave nekukasira kuita yekusagadzika nzira yekudzidzisa yakavhurika mamodheru ekufunga, uye maLab ari kudzokorora panzvimbo dzayo dzisina simba. Vatsvakurudzi vari kuongorora zvigadziriso zvehurefu uye kuoma kwakarerekera (zvakadai saDr. GRPO), token-level pane kutevedzana-level normalization, uye kubvisa kana kugadzirisa zvakare izwi reKL. Tarisira kubatanidzwa kwakasimba nemibairo inovimbika (masvomhu, kodhi, kushandisa chishandiso), kubata zvirinani kwemasaini mashoma, uye mahybrids anosanganisa mabhesi eboka nevatsoropodzi vasina kuremerwa kune agent, akawanda-nhanho mabasa.
Kuzvishandisa
Nei zvichikosha
- Mutengo uye bhajeti. Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
- Sarudzo dzakajeka. Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
- Kudzora kwemhando yepamusoro. Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.
Chenjera
- Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
- Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
- Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.
Matanho anotevera
- Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
- Benchmark pasi pechokwadi mutoro uye data mamiriro.
- Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
- Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.
Ramba Uchiongorora
Tsamba yemahara
Tora iyo yemazuva ese AI pfupiso
Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.
Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.
Zviedze iwe pachako
Take the Group Relative Policy Optimization quiz
Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.
Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo
Mibvunzo inowanzo bvunzwa
Chii chinonzi Group Relative Policy Optimization?
Boka Relative Policy Optimization (GRPO) inzira yekusimbisa-yekudzidza yemhando dzemitauro inotonga mhinduro yega yega vachipokana neboka rehama mhinduro kune imwecheteyo kukurumidza, kubvisa yakaparadzana kukosha network inoshandiswa nePPO. Yakava nemukurumbira seyakanyanya kudzidzisa trick kuseri kweDeepSeek's kufunga modhi.
Which separately learned estimator does the original GRPO approach avoid training?
GRPO uses a group-relative baseline rather than a separately learned value estimator. The size and resource savings depend on the PPO implementation used for comparison.
GRPO inoverengera sei mukana wekupedzisa kwakapihwa?
Kupedza kwega kwega kwakanakira (mubairo - boka zvinoreva) / chiyero cheboka kutsauka, saka boka remhinduro kune imwecheteyo kukurumidza rinoita seyekutanga.
Ndeapi mamodheru akaita kuti GRPO izivikanwe?
GRPO yakaunzwa uye yakakurumbira neDeepSeek, kunyanya muDeepSeekMath uye seyinjini yeRL kuseri kweDeepSeek-R1 yekufunga modhi.
Ndeipi basa rinoitwa neKL-divergence term muGRPO?
Chirango cheKL chinopesana nereferensi (kazhinji pre-RL) modhi inogadzika kudzidziswa kuitira kuti mutemo uvandudze pasina kudonha kana kukukurwa kuita zvinobuda.
Nei GRPO ichinyanya kukwezva pakudzidzisa mamodheru ekufunga pasvomhu kana kodhi?
Sampling mhinduro dzakawanda nekudzipa zvibodzwa zvine otomatiki kutarisisa mapeya zvakachena neGRPO's boka-yakajairika mabhenefiti, hapana anotsoropodza anodiwa.
Ramba uchidzidza
Mazano anoenderana
Mamwe madhairekitori akasarudzirwa nyaya iyi