Gukwirakwiza Politiki Yerekeye Amatsinda
Itsinda Rifitanye isano na Politiki Optimisiyoneri (GRPO) nuburyo bwo gushimangira-kwiga uburyo bwo guhuza neza imvugo yerekana imvugo igenzura buri gisubizo kubitsinda ryibisubizo byabavandimwe kubibazo bimwe, bikuraho umuyoboro wihariye wakoreshejwe na PPO.
Incamake
It became famous as the core training trick behind DeepSeek's reasoning models.
Kwibira cyane
GRPO ni variant ya politike-igenda ishimangira imyigire igamije gukora kugirango RL ihuze neza imiterere yindimi nini zihendutse kandi zihamye. PPO isanzwe ikenera 'kunegura' (icyitegererezo cyagaciro), hafi nini nka politiki ubwayo, kugirango igereranye uburyo buri kimenyetso ari cyiza. GRPO ikuraho uwo kunegura burundu. Kuri buri gisubizo cyerekana itsinda ryuzuza (vuga 8-64), ritsindire byose hamwe nibimenyetso byigihembo, hanyuma ubare ibyiza byose birangiye muguhuza ibihembo byacyo muburyo butandukanye bwo gutandukana kwitsinda. Hejuru-ugereranije ibisubizo birashimangirwa kandi munsi yikigereranyo cyahagaritswe. Ijambo KL-gutandukana rigumisha icyitegererezo hafi ya politiki yerekana. Yinjijwe na DeepSeek, yakoresheje DeepSeekMath hamwe na Moderi yo gutekereza ya DeepSeek-R1.
Ubushishozi
Igitekerezo cyingenzi ni ugusimbuza PPO yize agaciro shingiro hamwe nitsinda rya Monte Carlo. Kubitsinda ryibisubizo hamwe nibihembo r_i, buri nyungu ni A_i = (r_i - bivuze (r)) / std (r). Ayo manota asanzwe agwiza ibipimo byagabanijwe, neza nko muri PPO, hamwe na KL igihano cyerekeranye nicyitegererezo cyahagaritswe curbs drift. Kuberako ntamunenga watojwe, kwibuka no kubara hafi kimwe cya kabiri, kandi kuri buri kintu gisanzwe gitanga bisanzwe byapimwe, ibyiza-bitandukanye.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Ejo hazaza h'amatsinda afitanye isano na politiki
GRPO yahindutse byihuse uburyo busanzwe bwo guhugura uburyo bwo gutekereza bwifunguye, kandi laboratoire zisubiramo ahantu hakeye. Abashakashatsi barimo gushakisha ibyakosowe kuburebure no kubogama (nka Dr. GRPO), urwego rwikimenyetso aho kuba urwego rusanzwe rusanzwe, no gukuraho cyangwa guhindura ijambo KL. Tegereza kwishyira hamwe hamwe nibihembo bifatika (imibare, kode, gukoresha ibikoresho), gufata neza ibimenyetso bidakunze kubaho, hamwe na Hybride ihuza imirongo yibanze hamwe nabanegura koroheje kubikorwa, intambwe nyinshi.
Gushyira mu bikorwa Isi
Guhugura DeepSeek-R1 na DeepSeekMath kubyara urunigi rurerure-rwibitekerezo ukoresheje ibihembo bishingiye kumategeko yo gukosora kubibazo byimibare
Kuringaniza neza-ibisekuru-bigezweho aho buri gisubizo cyatanzwe gitangwa niba cyatsinze ibizamini, kandi itsinda risanzwe ryo gutora abatsinze
Gufungura-isoko ya RLHF imiyoboro (urugero, muri TRL na verl isomero) ukoresheje GRPO guhuza moderi zo kuganira utishyuye umuyoboro wihariye.
Gutezimbere amabwiriza-akurikira cyangwa imyitwarire yumutekano muguhitamo ibisubizo byinshi kuri buri gihe no guhemba abo ibihembo byikitegererezo cyo hejuru ugereranije na bagenzi babo
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Group Relative Policy Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Gukwirakwiza Politiki Yegeranye
Ibibazo bikunze kubazwa
What is Group Relative Policy Optimization?
Itsinda Rifitanye isano na Politiki Optimisiyoneri (GRPO) nuburyo bwo gushimangira-kwiga uburyo bwo guhuza neza imvugo yerekana imvugo igenzura buri gisubizo kubitsinda ryibisubizo byabavandimwe kubibazo bimwe, bikuraho umuyoboro wihariye wakoreshejwe na PPO. Yabaye icyamamare nkamahugurwa yibanze inyuma yibitekerezo bya DeepSeek.
Ni ibihe bintu by'ingenzi bigize PPO GRPO ikuraho?
Guhindura umukono wa GRPO ni uguta PPO yize agaciro / kunegura icyitegererezo, mubisanzwe bingana na politiki, bizigama kwibuka cyane na compte.
Nigute GRPO ibara inyungu zo kurangiza gutangwa?
Buri nyungu yo kurangiza ni (ibihembo - itsinda bisobanura) / itsinda risanzwe gutandukana, bityo itsinda ryibisubizo kubintu bimwe byihutirwa bikora nkibanze.
Ni ubuhe bwoko bwatumye GRPO imenyekana neza?
GRPO yatangijwe kandi ikundwa na DeepSeek, cyane cyane muri DeepSeekMath ndetse nka moteri ya RL inyuma yicyitegererezo cya DeepSeek-R1.
Ni uruhe ruhare ijambo KL-gutandukana rifite muri GRPO?
Igihano cya KL cyerekana icyerekezo (mubisanzwe mbere ya RL) gisanzwe gihugura amahugurwa kugirango politiki itere imbere nta gusenyuka cyangwa gutembera mubisubizo byangiritse.
Kuki GRPO ishimishije cyane mumahugurwa yo gutekereza kubitekerezo cyangwa kode?
Gutoranya ibisubizo byinshi no kubitsindira hamwe no gukosora byikora kugenzura byombi hamwe neza hamwe nitsinda rya GRPO risanzwe, nta kunegura bikenewe.