Ubuyobozi bwa tekiniki

Gukwirakwiza Politiki Yerekeye Amatsinda

Itsinda Rifitanye isano na Politiki Optimisiyoneri (GRPO) nuburyo bwo gushimangira-kwiga uburyo bwo guhuza neza imvugo yerekana imvugo igenzura buri gisubizo kubitsinda ryibisubizo byabavandimwe kubibazo bimwe, bikuraho umuyoboro wihariye wakoreshejwe na PPO.

Umva iki gitaboAI-byavuzwe · ~2 min

Kuriyi page2 min soma
  1. Incamake
  2. Kwibira cyane
  3. Gushyira mu bikorwa Isi
  4. Ejo hazaza h'amatsinda afitanye isano na politiki
  5. Kubishyira mubikorwa
  6. Komeza Ubushakashatsi
  7. Ibibazo bikunze kubazwa

Incamake

It became famous as the core training trick behind DeepSeek's reasoning models.

Kwibira cyane

GRPO is a variant of policy-gradient reinforcement learning designed to make RL fine-tuning of large language models cheaper and more stable. Many PPO implementations train a value model to estimate returns; its size and cost depend on the implementation. GRPO removes that critic entirely. For each prompt it samples a group of completions (say 8-64), scores them all with a reward signal, and then computes each completion's advantage by standardizing its reward against the group's mean and standard deviation. Above-average answers are reinforced and below-average ones suppressed. A KL-divergence term keeps the model close to a reference policy. Introduced by DeepSeek, it powered DeepSeekMath and the DeepSeek-R1 reasoning models.

Gushyira mu bikorwa Isi

Guhugura DeepSeek-R1 na DeepSeekMath kubyara urunigi rurerure-rwibitekerezo ukoresheje ibihembo bishingiye kumategeko yo gukosora kubibazo byimibare

Kuringaniza neza-ibisekuru-bigezweho aho buri gisubizo cyatanzwe gitangwa niba cyatsinze ibizamini, kandi itsinda risanzwe ryo gutora abatsinze

Gufungura-isoko ya RLHF imiyoboro (urugero, muri TRL na verl isomero) ukoresheje GRPO guhuza moderi zo kuganira utishyuye umuyoboro wihariye.

Gutezimbere amabwiriza-akurikira cyangwa imyitwarire yumutekano muguhitamo ibisubizo byinshi kuri buri gihe no guhemba abo ibihembo byikitegererezo cyo hejuru ugereranije na bagenzi babo

Ejo hazaza h'amatsinda afitanye isano na politiki

GRPO yahindutse byihuse uburyo busanzwe bwo guhugura uburyo bwo gutekereza bwifunguye, kandi laboratoire zisubiramo ahantu hakeye. Abashakashatsi barimo gushakisha ibyakosowe kuburebure no kubogama (nka Dr. GRPO), urwego rwikimenyetso aho kuba urwego rusanzwe rusanzwe, no gukuraho cyangwa guhindura ijambo KL. Tegereza kwishyira hamwe hamwe nibihembo bifatika (imibare, kode, gukoresha ibikoresho), gufata neza ibimenyetso bidakunze kubaho, hamwe na Hybride ihuza imirongo yibanze hamwe nabanegura koroheje kubikorwa, intambwe nyinshi.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
  • Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
  • Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.

Witondere

  • Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
  • Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
  • Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.

Intambwe ikurikira

  1. Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
  2. Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
  3. Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
  4. Tegura inzira yo gusubiza ibyabaye mbere yo gupima.

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the Group Relative Policy Optimization quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

What is Group Relative Policy Optimization?

Itsinda Rifitanye isano na Politiki Optimisiyoneri (GRPO) nuburyo bwo gushimangira-kwiga uburyo bwo guhuza neza imvugo yerekana imvugo igenzura buri gisubizo kubitsinda ryibisubizo byabavandimwe kubibazo bimwe, bikuraho umuyoboro wihariye wakoreshejwe na PPO. Yabaye icyamamare nkamahugurwa yibanze inyuma yibitekerezo bya DeepSeek.

Ni ubuhe buryo bwo kugereranya bwizewe ukwabo uburyo bw'umwimerere bwa GRPO bwirinda kwigisha?

GRPO uses a group-relative baseline rather than a separately learned value estimator. The size and resource savings depend on the PPO implementation used for comparison.

Nigute GRPO ibara inyungu zo kurangiza gutangwa?

Buri nyungu yo kurangiza ni (ibihembo - itsinda bisobanura) / itsinda risanzwe gutandukana, bityo itsinda ryibisubizo kubintu bimwe byihutirwa bikora nkibanze.

Ni ubuhe bwoko bwatumye GRPO imenyekana neza?

GRPO yatangijwe kandi ikundwa na DeepSeek, cyane cyane muri DeepSeekMath ndetse nka moteri ya RL inyuma yicyitegererezo cya DeepSeek-R1.

Ni uruhe ruhare ijambo KL-gutandukana rifite muri GRPO?

Igihano cya KL cyerekana icyerekezo (mubisanzwe mbere ya RL) gisanzwe gihugura amahugurwa kugirango politiki itere imbere nta gusenyuka cyangwa gutembera mubisubizo byangiritse.

Kuki GRPO ishimishije cyane mumahugurwa yo gutekereza kubitekerezo cyangwa kode?

Gutoranya ibisubizo byinshi no kubitsindira hamwe no gukosora byikora kugenzura byombi hamwe neza hamwe nitsinda rya GRPO risanzwe, nta kunegura bikenewe.