HAGAHA Farsamada
Horumarinta Siyaasadda Qaraabada ee Kooxda
Hagaajinta Siyaasadda Qaraabada Kooxda (GRPO) waa habka xoojinta-waxbarashada ee habaynta hab-dhaqanka hab-luuqadeed kaas oo ku qiimeeya jawaab kasta oo ka dhan ah koox jawaabaha walaalaha ah isla markiiba, meeshana ka saaraysa isku xidhka qiimaha gaarka ah ee ay isticmaasho PPO.
Dhageyso hagahanAI-waa laga sheekeeyay · ~2 min
Boggaan2 daqiiqo akhri
Dulmar
It became famous as the core training trick behind DeepSeek's reasoning models.
quusid qoto dheer
GRPO is a variant of policy-gradient reinforcement learning designed to make RL fine-tuning of large language models cheaper and more stable. Many PPO implementations train a value model to estimate returns; its size and cost depend on the implementation. GRPO removes that critic entirely. For each prompt it samples a group of completions (say 8-64), scores them all with a reward signal, and then computes each completion's advantage by standardizing its reward against the group's mean and standard deviation. Above-average answers are reinforced and below-average ones suppressed. A KL-divergence term keeps the model close to a reference policy. Introduced by DeepSeek, it powered DeepSeekMath and the DeepSeek-R1 reasoning models.
Dhaqangelinta Adduunka-dhabta ah
Tababarka DeepSeek-R1 iyo DeepSeekMath si ay u soo saaraan silsilad-fikrad dheer oo fikrado ah iyadoo la adeegsanayo abaal-marinnada saxda ah ee qaanuunka ku salaysan ee dhibaatooyinka xisaabta
Qaababka jiilka code-ka-habaynta wanagsan halkaaso xal kasta oo muunad ah lagu qiimeeyay haddi ay ku gudubto imtixaanada cutubka, kooxdana ay caadi tahay in ay doortaan guulaystayaasha
Tubooyinka RLHF-ka furan (tusaale, TRL iyo maktabadaha verl) iyadoo la isticmaalayo GRPO si loo waafajiyo moodooyinka wada sheekaysiga adoon bixinayn shabakad qiime leh
Hagaajinta hab-dhaqanka edbinta ama hab-dhaqanka badbaadada iyadoo la tijaabinayo dhawr jawaabo isla markiiba lana abaalmariyo kuwa abaalgudka qiimaha ugu sarreeya marka loo eego asaagooda
Mustaqbalka Siyaasadda Qaraabada ee Kooxda
GRPO waxa ay si degdeg ah u noqotay cunto karinta caadiga ah ee tababbarka moodooyinka caqliga furan, shaybaadhkana waxa ay ku celcelinayaan meelaha daciifka ah. Cilmi baadhayaashu waxay sahamiyaan hagaajinta dhererka iyo eexda dhibka (sida Dr. GRPO), heerka calaamada halkii ay ka ahaan lahaayeen isku xigxiga caadiga ah, iyo ka saarida ama dib u qaabaynta ereyga KL. Filo is dhexgal adag oo leh abaal-marin la xaqiijin karo (xisaab, koodhka, isticmaalka aaladda), maaraynta ka wanaagsan ee calaamadaha aan fiicneyn, iyo isku-dhafka isku-dhafka ah ee isku-dhafka kooxda iyo dhaleeceynta fudud ee hawl-qabad, hawlo badan.
Ku dhaqangelinta
Maxay muhiim u tahay
- Qiimaha iyo miisaaniyada. Go'aamada qaab-dhismeedku waxay horseedaan waxqabadka iyo kharashka hawlgalka sannadaha.
- Go'aamo cad. Waxbarashada farsamada waxay ka caawisaa kooxaha inay doortaan xidhmo sax ah, ma aha oo kaliya kan ugu cusub.
- Xakamaynta tayada. Doorashooyinka injineernimada ee wanaagsan waxay yareeyaan shilalka la isku halleyn karo ee wax soo saarka.
Iska ilaali
- Hagaajinta hal bartilmaameed waxay qarin kartaa daciifnimada nidaamka ballaaran.
- Kaabayaasha dhaqaalaha iyo dayactirka inta badan waa la dhayalsadaa.
- Nabadgelyada iyo daldaloolada u fiirsashada ayaa kori kara marka nidaamyadu noqdaan kuwo aad u adag.
Tallaabooyinka xiga
- Qeex daahida, tayada, iyo bartilmaameedyada qiimaha ka hor inta aan la hirgelin.
- Benchmark marka la eego culeyska dhabta ah iyo xaaladaha xogta.
- La socodka qalabka khaladaadka, leexashada, iyo saamaynta isticmaalaha.
- U diyaari dib-u-noqoshada iyo dariiqyada jawaab-celinta dhacdada ka hor inta aanad miisaan.
Sii wad Sahaminta
Warqad bilaash ah
Hel warbixinta maalinlaha ah ee AI
Saddex sheeko oo la xaqiijiyay oo AI ah maalin kasta oo shaqo ah, oo lagu qoray Ingiriis fudud. Bilaash had iyo jeer, aan xayaysiis lahayn.
Hal iimayl maalin kasta oo shaqo ah. Ka bax hal gujin. Marna ma iibinno ama ma wadaagno ciwaankaaga.
Is tijaabi
Take the Group Relative Policy Optimization quiz
Fikrad degdeg ah oo ku saabsan jawaab kasta, iyo shahaado la wadaajin karo oo leh aqoonsi la xaqiijin karo markaad ka gudubto koorso.
Taageer waxbarashada AI bilaashka ah. AI Understanding waa 501(c)(3) aan faa'iido doon ahayn — xayaysiis la'aan, albaab lacag ah ma jiro, had iyo jeer. Samee deeq
Su'aalaha soo noqnoqda
What is Group Relative Policy Optimization?
Hagaajinta Siyaasadda Qaraabada Kooxda (GRPO) waa habka xoojinta-waxbarashada ee habaynta hab-dhaqanka hab-luuqadeed kaas oo ku qiimeeya jawaab kasta oo ka dhan ah koox jawaabaha walaalaha ah isla markiiba, meeshana ka saaraysa isku xidhka qiimaha gaarka ah ee ay isticmaasho PPO. Waxay caan ku noqotay khiyaamada tababarka xudunta u ah moodooyinka sababaynta DeepSeek.
Which separately learned estimator does the original GRPO approach avoid training?
GRPO uses a group-relative baseline rather than a separately learned value estimator. The size and resource savings depend on the PPO implementation used for comparison.
Sidee GRPO u xisaabisaa faa'iidada dhamaystirka la siiyay?
Dhammaystir kasta faa'iidadiisa waa (abaal-marin-koox celcelis ahaan) / weecasho heerka kooxeed, markaa kooxda jawaabaha isla markiiba waxay u dhaqmaan sida gundhigga.
Qaabkee ayaa GRPO si fiican u yaqaan?
GRPO waxaa soo bandhigay oo caan ku ah DeepSeek, gaar ahaan DeepSeekMath iyo sida matoorka RL ee ka dambeeya moodooyinka sababaynta DeepSeek-R1.
Doorkee ayuu ereyga KL-kala-duwanaanshaha ku leeyahay GRPO?
Ciqaabta KL ee ka dhanka ah tixraaca (sida caadiga ah ka hor-RL) moodelku waxa uu joogtaa tababarka si siyaasaddu u horumarto iyada oo aan la dumin ama u leexin wax soo saar xumaaday.
Waa maxay sababta GRPO ay si gaar ah ugu soo jiidanayso moodooyinka sababaynta tababarka ee xisaabta ama koodka?
Samaynta xalal badan iyo ku-sixinta iyaga oo si toos ah u hubinaya saxnaanta tooska ah ee lammaaneyaasha si nadiif ah faa'iidooyinka caadiga ah ee kooxda GRPO, dhaleecayn looma baahna.
Sii wad waxbarashada
Tilmaamaha la xidhiidha
Tilmaamayaal badan ayaa loo doortay mawduucan