Ukuthuthukiswa Kwenqubomgomo Ehlobene Neqembu
I-Group Relative Policy Optimization (GRPO) iyindlela yokuqinisa yokufunda yamamodeli olimi ashuna kahle ahlulela impendulo ngayinye ngokumelene neqembu lezimpendulo eziyizelamani ekwazisweni okufanayo, esusa inethiwekhi yenani ehlukile esetshenziswa i-PPO.
Uhlolojikelele
It became famous as the core training trick behind DeepSeek's reasoning models.
I-Deep Dive
I-GRPO iwuhlobo oluhlukile lokufunda okuqiniswa kwenqubomgomo-gradient okudizayinelwe ukwenza ukulungisa kahle kwe-RL kwamamodeli wezilimi ezinkulu kushibhe futhi kuzinze kakhudlwana. I-PPO evamile idinga 'umgxeki' ofundiwe (imodeli yenani), cishe enkulu njengenqubomgomo ngokwayo, ukuze ilinganisele ukuthi ithokheni ngalinye lihle kangakanani. I-GRPO isusa lowo mgxeki ngokuphelele. Emyalweni ngamunye yenza amasampula eqembu lokuqedela (ithi 8-64), iwathole wonke ngesignali yomklomelo, bese ihlanganisa inzuzo yokuqeda ngakunye ngokumisa umvuzo wayo ngokumelene nesilinganiso nokuchezuka okujwayelekile kweqembu. Izimpendulo ezingaphezu kwesilinganiso ziyaqiniswa futhi ezingaphansi kwe-avareji ziyacindezelwa. Itemu le-KL-divergence ligcina imodeli iseduze nenqubomgomo yereferensi. Yethulwe yi-DeepSeek, inikeze amandla i-DeepSeekMath kanye nemodeli ye-DeepSeek-R1 yokucabanga.
I-Technical Insight
Umbono obalulekile ukufaka isisekelo senani le-PPO esikhundleni sesisekelo seqembu le-Monte Carlo. Eqenjini lemiphumela enemiklomelo engu-r_i, inzuzo ngayinye ithi A_i = (r_i - mean(r)) / std(r). Lowo mphumela ojwayelekile uphindaphinda isilinganiso samathuba asikiwe, ncamashi njengaku-PPO, kanye nenhlawulo ye-KL ngokumelene nemodeli yereferensi efriziwe inqamula ukukhukhuleka. Ngoba akekho umgxeki oqeqeshiwe, inkumbulo nokubala cishe kuhhafu, futhi ukujwayela ngokushesha kunikeza izinzuzo ezilinganiselwe ngokwemvelo, zokuhlukahluka okuphansi.
I-Strategic Impact
Izindleko kanye nesabelomali
Izinqumo zezakhiwo ziqhuba ukusebenza kanye nezindleko zokusebenza iminyaka.
Izinqumo ezicacile
Imfundo yobuchwepheshe isiza amaqembu ukuthi akhethe isitaki esifanele, hhayi nje esisha.
Ukulawulwa kwekhwalithi
Izinketho ezingcono zobunjiniyela zinciphisa izehlakalo ezinokwethenjelwa ekukhiqizeni.
Ikusasa Lokuthuthukiswa Kwenqubomgomo Yeqembu
I-GRPO isiphenduke iresiphi ezenzakalelayo yokuqeqesha amamodeli avulekile okucabanga, futhi amalebhu aphindaphinda ezindaweni zawo ezibuthakathaka. Abacwaningi bahlola ukulungiswa kokuchema kobude nobunzima (okufana noDkt. GRPO), ileveli yamathokheni esikhundleni sokujwayelekile kwezinga lokulandelana, nokususa noma ukulolonga kabusha igama le-KL. Lindela ukuhlanganiswa okuqinile okunemiklomelo eqinisekiswa (izibalo, ikhodi, ukusetshenziswa kwamathuluzi), ukuphatha kangcono amasignali agqagqene, namahybrids ahlanganisa isisekelo seqembu nabagxeki abangasindi bemisebenzi, yezinyathelo eziningi.
Ukuqaliswa Komhlaba Wangempela
Ukuqeqesha i-DeepSeek-R1 kanye ne-DeepSeekMath ukuze kukhiqizwe uchungechunge olude lokucabanga usebenzisa imivuzo esekelwe emthethweni yokunemba ezinkingeni zezibalo.
Amamodeli okukhiqiza ikhodi yokuhlela kahle lapho isixazululo ngasinye esiyisampula sitholwa ngokuthi siyaphumelela yini ekuhlolweni kweyunithi, futhi iqembu lenziwa ngokwejwayelekile ukuze likhethe abawinile.
Amapayipi omthombo ovulekile we-RLHF (isb., ku-TRL kanye namalabhulali e-verl) kusetshenziswa i-GRPO ukuqondisa amamodeli engxoxo ngaphandle kokukhokhela inethiwekhi yenani elihlukile
Ukuthuthukisa ukulandela imiyalelo noma ukuziphatha kokuphepha ngokuthatha isampula izimpendulo ezimbalwa ngesikhathi ngasinye kanye nokuklomelisa lezo imodeli yomklomelo amanani aphezulu uma kuqhathaniswa nontanga yabo.
Izingozi & Guardrails
Ukuthuthukisa ibhentshimakhi eyodwa kungafihla ubuthakathaka obubanzi besistimu.
Izindleko zengqalasizinda nezokulungisa zivame ukubukelwa phansi.
Izikhala zokuphepha nokubonakala zingakhula njengoba izinhlelo ziba nzima kakhulu.
Ukuqalisa Umhlahlandlela
Chaza ukubambezeleka, ikhwalithi, nezindleko ezihlosiwe ngaphambi kokuqaliswa.
Ibhentshimakhi ngaphansi komthwalo wangempela nezimo zedatha.
Ukuqapha amathuluzi amaphutha, ukukhukhuleka, nomthelela wabasebenzisi.
Lungiselela izindlela zokuhlehlisa nezigameko ngaphambi kokukala.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Group Relative Policy Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
I-Proximal Policy Optimization
Imibuzo evame ukubuzwa
What is Group Relative Policy Optimization?
I-Group Relative Policy Optimization (GRPO) iyindlela yokuqinisa yokufunda yamamodeli olimi ashuna kahle ahlulela impendulo ngayinye ngokumelene neqembu lezimpendulo eziyizelamani ekwazisweni okufanayo, esusa inethiwekhi yenani ehlukile esetshenziswa i-PPO. Kwaduma njengeqhinga lokuqeqesha eliyisisekelo ngemuva kwamamodeli okucabanga we-DeepSeek.
Iyiphi ingxenye enkulu ye-PPO i-GRPO eyiqedayo?
Ushintsho lwesiginesha ye-GRPO lwehlisa inani elifundiwe/imodeli yokugxeka ye-PPO, evamise ukuba nosayizi ofanayo nowenqubomgomo, ilondoloza inkumbulo enkulu nokubala.
I-GRPO ihlanganisa kanjani inzuzo ngokuqedwa okunikeziwe?
Inzuzo yokuqeda ngakunye iwukuba (umvuzo - kusho iqembu) / ukuchezuka okujwayelekile kweqembu, ngakho iqembu lezimpendulo ekwazisweni okufanayo lisebenza njengesisekelo.
Imaphi amamodeli enze i-GRPO yaziwe?
I-GRPO yethulwa futhi yaduma ngabakwa-DeepSeek, ikakhulukazi ku-DeepSeekMath nanjengenjini ye-RL ngemuva kwemodeli yokucabanga ye-DeepSeek-R1.
Iyiphi indima edlalwa yi-KL-divergence term ku-GRPO?
Inhlawulo ye-KL ngokumelene nereferensi (imvamisa imodeli yangaphambi kwe-RL) yenza ukuqeqeshwa kujwayelekile ukuze inqubomgomo ibe ngcono ngaphandle kokuwa noma ukukhukhuleka emiphumeleni ewohlokayo.
Kungani i-GRPO ikhanga ikakhulukazi ekuqeqesheni amamodeli okucabanga ngezibalo noma ikhodi?
Ukusampula izisombululo eziningi nokuzifaka ngokunemba okuzenzakalelayo kuhlola amapheya ngokuhlanzekile ngezinzuzo ezijwayelekile zeqembu le-GRPO, akekho umgxeki odingekayo.