HejuruUbuyobozi bukurikira
Gukwirakwiza Politiki Yegeranye
Ururimi AI
Ubuyobozi bwa tekiniki
Itsinda Rifitanye isano na Politiki Optimisiyoneri (GRPO) nuburyo bwo gushimangira-kwiga uburyo bwo guhuza neza imvugo yerekana imvugo igenzura buri gisubizo kubitsinda ryibisubizo byabavandimwe kubibazo bimwe, bikuraho umuyoboro wihariye wakoreshejwe na PPO.
Umva iki gitaboAI-byavuzwe · ~2 min
It became famous as the core training trick behind DeepSeek's reasoning models.
GRPO is a variant of policy-gradient reinforcement learning designed to make RL fine-tuning of large language models cheaper and more stable. Many PPO implementations train a value model to estimate returns; its size and cost depend on the implementation. GRPO removes that critic entirely. For each prompt it samples a group of completions (say 8-64), scores them all with a reward signal, and then computes each completion's advantage by standardizing its reward against the group's mean and standard deviation. Above-average answers are reinforced and below-average ones suppressed. A KL-divergence term keeps the model close to a reference policy. Introduced by DeepSeek, it powered DeepSeekMath and the DeepSeek-R1 reasoning models.
Guhugura DeepSeek-R1 na DeepSeekMath kubyara urunigi rurerure-rwibitekerezo ukoresheje ibihembo bishingiye kumategeko yo gukosora kubibazo byimibare
Kuringaniza neza-ibisekuru-bigezweho aho buri gisubizo cyatanzwe gitangwa niba cyatsinze ibizamini, kandi itsinda risanzwe ryo gutora abatsinze
Gufungura-isoko ya RLHF imiyoboro (urugero, muri TRL na verl isomero) ukoresheje GRPO guhuza moderi zo kuganira utishyuye umuyoboro wihariye.
Gutezimbere amabwiriza-akurikira cyangwa imyitwarire yumutekano muguhitamo ibisubizo byinshi kuri buri gihe no guhemba abo ibihembo byikitegererezo cyo hejuru ugereranije na bagenzi babo
GRPO yahindutse byihuse uburyo busanzwe bwo guhugura uburyo bwo gutekereza bwifunguye, kandi laboratoire zisubiramo ahantu hakeye. Abashakashatsi barimo gushakisha ibyakosowe kuburebure no kubogama (nka Dr. GRPO), urwego rwikimenyetso aho kuba urwego rusanzwe rusanzwe, no gukuraho cyangwa guhindura ijambo KL. Tegereza kwishyira hamwe hamwe nibihembo bifatika (imibare, kode, gukoresha ibikoresho), gufata neza ibimenyetso bidakunze kubaho, hamwe na Hybride ihuza imirongo yibanze hamwe nabanegura koroheje kubikorwa, intambwe nyinshi.
Ibaruwa y'amakuru y'ubuntu
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
Itsinda Rifitanye isano na Politiki Optimisiyoneri (GRPO) nuburyo bwo gushimangira-kwiga uburyo bwo guhuza neza imvugo yerekana imvugo igenzura buri gisubizo kubitsinda ryibisubizo byabavandimwe kubibazo bimwe, bikuraho umuyoboro wihariye wakoreshejwe na PPO. Yabaye icyamamare nkamahugurwa yibanze inyuma yibitekerezo bya DeepSeek.
GRPO uses a group-relative baseline rather than a separately learned value estimator. The size and resource savings depend on the PPO implementation used for comparison.
Buri nyungu yo kurangiza ni (ibihembo - itsinda bisobanura) / itsinda risanzwe gutandukana, bityo itsinda ryibisubizo kubintu bimwe byihutirwa bikora nkibanze.
GRPO yatangijwe kandi ikundwa na DeepSeek, cyane cyane muri DeepSeekMath ndetse nka moteri ya RL inyuma yicyitegererezo cya DeepSeek-R1.
Igihano cya KL cyerekana icyerekezo (mubisanzwe mbere ya RL) gisanzwe gihugura amahugurwa kugirango politiki itere imbere nta gusenyuka cyangwa gutembera mubisubizo byangiritse.
Gutoranya ibisubizo byinshi no kubitsindira hamwe no gukosora byikora kugenzura byombi hamwe neza hamwe nitsinda rya GRPO risanzwe, nta kunegura bikenewe.
Komeza wige
Abandi bayobozi batoranijwe kuriyi ngingo
HejuruUbuyobozi bukurikira
Gukwirakwiza Politiki Yegeranye
Ururimi AI