Boka Relative Policy Optimization
Boka Relative Policy Optimization (GRPO) inzira yekusimbisa-yekudzidza yemhando dzemitauro inotonga mhinduro yega yega vachipokana neboka rehama mhinduro kune imwecheteyo kukurumidza, kubvisa yakaparadzana kukosha network inoshandiswa nePPO.
Pfupiso
It became famous as the core training trick behind DeepSeek's reasoning models.
Kudzika Kwakadzika
GRPO musiyano wepolicy-gradient kusimbisa kudzidza kwakagadzirirwa kuita RL kugadzirisa zvakanaka kwemhando dzemitauro mikuru kudhura uye kugadzikana. PPO yakajairwa inoda akadzidza 'anotsoropodza' (value modhi), zvakakura segwaro racho pacharo, kufungidzira kunaka kwechiratidzo chega chega. GRPO inobvisa mutsoropodzi uyu zvachose. Pakuchimbidza kwega kwega inoisa sampuli boka rekupedzisa (inoti 8-64), inovaverengera vese nechiratidzo chemubairo, uyezve inoverengera mukana wekupedzisa kwega kwega nekuenzanisa mubairo wayo zvichienderana nekurevesa kweboka uye kutsauka kwakajairwa. Pamusoro-avhareji mhinduro dzinosimbiswa uye dzepasi-avhareji dzakatsikirirwa. Izwi reKL-divergence rinochengeta modhi iri padyo nereferensi mutemo. Yakaunzwa neDeepSeek, yakapa simba DeepSeekMath uye iyo DeepSeek-R1 yekufunga modhi.
Technical Insight
Pfungwa yakakosha kutsiva PPO's yakadzidza kukosha baseline neMonte Carlo boka yekutanga. Kune boka rezvabuda nemubairo r_i, mukana wega wega A_i = (r_i - zvinoreva (r)) / std(r). Icho chibodzwa chakajairwa chinowedzera chiyero chakachekwa, sezvakaita muPPO, uye chirango cheKL chinopesana nechando chereferensi modhi curbs kudonha. Nekuti hapana mutsoropodzi akadzidziswa, ndangariro uye compute ingangoita hafu, uye iyo-yekukurumidza kusimudzira inopa yakasarudzika yakayerwa, yakaderera-musiyano mabhenefiti.
Strategic Impact
Mutengo uye bhajeti
Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
Sarudzo dzakajeka
Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
Kudzora kwemhando yepamusoro
Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.
Ramangwana reBoka Relative Policy Optimization
GRPO yave nekukasira kuita yekusagadzika nzira yekudzidzisa yakavhurika mamodheru ekufunga, uye maLab ari kudzokorora panzvimbo dzayo dzisina simba. Vatsvakurudzi vari kuongorora zvigadziriso zvehurefu uye kuoma kwakarerekera (zvakadai saDr. GRPO), token-level pane kutevedzana-level normalization, uye kubvisa kana kugadzirisa zvakare izwi reKL. Tarisira kubatanidzwa kwakasimba nemibairo inovimbika (masvomhu, kodhi, kushandisa chishandiso), kubata zvirinani kwemasaini mashoma, uye mahybrids anosanganisa mabhesi eboka nevatsoropodzi vasina kuremerwa kune agent, akawanda-nhanho mabasa.
Real-World Implementation
Kudzidzisa DeepSeek-R1 uye DeepSeekMath kuburitsa refu ketani-ye-yekufunga kufunga uchishandisa mutemo-wakavakirwa kurongeka mibairo pamatambudziko esvomhu.
Yakanaka-tuning kodhi-chizvarwa modhi apo yega yega mhinduro inopihwa kana ichipfuura bvunzo dzeyuniti, uye boka rinojairwa kusarudza vanokunda.
Open-source RLHF mapaipi (semuenzaniso, muTRL uye verl maraibhurari) uchishandisa GRPO kurongedza mhando dzekutaura pasina kubhadhara kune yakaparadzana kukosha network.
Kuvandudza kuraira-kutevera kana kuchengetedza maitiro nekuenzanisa akati wandei mhinduro nekukasira uye kupa mubairo iwo mubairo wemubairo mitengo yepamusoro kune vezera ravo.
Njodzi & Guardrails
Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.
Implementation Roadmap
Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
Benchmark pasi pechokwadi mutoro uye data mamiriro.
Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.
Ramba Uchiongorora
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Group Relative Policy Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gaidhi rinotevera
Proximal Policy Optimization
Mibvunzo inowanzo bvunzwa
What is Group Relative Policy Optimization?
Boka Relative Policy Optimization (GRPO) inzira yekusimbisa-yekudzidza yemhando dzemitauro inotonga mhinduro yega yega vachipokana neboka rehama mhinduro kune imwecheteyo kukurumidza, kubvisa yakaparadzana kukosha network inoshandiswa nePPO. Yakava nemukurumbira seyakanyanya kudzidzisa trick kuseri kweDeepSeek's kufunga modhi.
Ndechipi chikamu chikuru chePPO chinobviswa neGRPO?
GRPO's siginecha shanduko iri kudonhedza PPO's yakadzidza kukosha / yeanotsoropodza modhi, iyo inowanzoita saizi yakafanana nepolicy, kuchengetedza yakakura ndangariro uye compute.
GRPO inoverengera sei mukana wekupedzisa kwakapihwa?
Kupedza kwega kwega kwakanakira (mubairo - boka zvinoreva) / chiyero cheboka kutsauka, saka boka remhinduro kune imwecheteyo kukurumidza rinoita seyekutanga.
Ndeapi mamodheru akaita kuti GRPO izivikanwe?
GRPO yakaunzwa uye yakakurumbira neDeepSeek, kunyanya muDeepSeekMath uye seyinjini yeRL kuseri kweDeepSeek-R1 yekufunga modhi.
Ndeipi basa rinoitwa neKL-divergence term muGRPO?
Chirango cheKL chinopesana nereferensi (kazhinji pre-RL) modhi inogadzika kudzidziswa kuitira kuti mutemo uvandudze pasina kudonha kana kukukurwa kuita zvinobuda.
Nei GRPO ichinyanya kukwezva pakudzidzisa mamodheru ekufunga pasvomhu kana kodhi?
Sampling mhinduro dzakawanda nekudzipa zvibodzwa zvine otomatiki kutarisisa mapeya zvakachena neGRPO's boka-yakajairika mabhenefiti, hapana anotsoropodza anodiwa.