Groupe bi gëna xéewale ci wàllu politik
Groupe Relative Policy Optimization (GRPO) xeetu jàng-jàng buy dooleel la ngir gëna suqali xeetu làkk yi, di àtte tontu bu nekk ci tontub mbokk yu bari ci benn laaj, di dindi reso valeur bu tàqaloo bi PPO di jëfandikoo.
Résumé
It became famous as the core training trick behind DeepSeek's reasoning models.
Plongeur bu xóot
GRPO benn xeetu njàngale la buy gëna dooleel politik, ñu defaree ko ngir gëna yombal RL ci xeetu làkk yu mag, gëna yomb te gëna dëgër. PPO buñ miin mingi soxla 'critique' bu jàng (modèle de valeur), lu tollu ci ni politik bi ci boppam, ngir xayma ni token bu nekk baax. GRPO dindi na ŋàññikat boobu lëmm. Ci laaj bu nekk dafay jël misaalu kuréel gu matt (nu wax 8-64), daal di leen jox poñ yépp ak siñaal neexal, ba noppi xayma njariñu matt bu nekk ci méngale neexal bi ak moyenn bu kuréel gi ak standard deviation. Tontu yi gëna kawe moyenne dañu leen di dooleel, tontu yi gëna suufe moyenne dañu leen di dindi. Terme KL-divergence dafay tëye model bi ci wetu politiku royuwaay. DeepSeek moo ko dugal, moo dooleel DeepSeekMath ak xeetu xalaat DeepSeek-R1.
Gis-gis xarala
Li gëna am solo mooy ñu wecci li PPO jàng ci valeur ak limu grupu Monte Carlo. Ngir kuréel gu génne ak neexal r_i, njariñ bu nekk mooy A_i = (r_i - moyenne (r)) / std (r). Boobu poñ buñ normalisé dafay yokk ratio probabilité biñ dagg, ni ci PPO, ak penalti KL ci kaw modelu royuwaay bu liw buy tere derivation. Ndax amul benn ŋàññikat buñu tàggat, mémoire ak calcul dañuy xaaj, te normalisasioŋ bu nekk dafay joxe njariñ yu bari te bariwul variance.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu Groupe Relatif Politik Optimisation
GRPO dafa gaaw nekk rëset buñ miin ngir tàggat xeetu xalaat yu ubbeeku yi, ba noppi laboratoire yi dañuy baamtu ci barab yu néew doole yi. Gëstukat yi ñu ngi jàngat saafara yi ngir guddaay ak jafe-jafe (lu melni Dr. GRPO), niveau token moo gën normalisation niveau sequence, ak dindi wala soppali terme KL. Xaarandi lëkkaloo bu gëna dëgër ak neexal yuñ mëna saytu (math, kode, jëfandikoo jumtukaay), gëna mëna jëfandikoo siñaal yu néew, ak ibrid yuy boole grupu baseline ak ŋàññikat yu woyof ngir liggéey yu bari, yu bari.
Doxal ci àdduna dëgg
Taggat DeepSeek-R1 ak DeepSeekMath ngir génne xalaat yu guddu ci jëfandikoo sàrt yu lalu ci njub ci neexal ci jafe-jafe math
Royuwaayi defar kode yu baax, fu bépp saafara buñ jël ñu jël poñ ci ndax jàll na test yuñ def, ba noppi ñu normalise kuréel gi ngir tànn ki jël raw-gàddu gi
Pipeline RLHF yu ubbeeku (lu melni, ci bibliotek TRL ak verl) di jëfandikoo GRPO ngir méngale xeetu waxtaan yi te doo fay beneen reso bu am valeur
Yokkatal topp tegtal yi wala jeffin ju wóor ci sampling tontu yu bari ci laaj bu nekk ak neexal ñooñu xeetu neexal bi gëna rëy ci seeni moroom
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Group Relative Policy Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Optimisation ci politik yu jege
Laaj yi ñuy faral di laaj
What is Group Relative Policy Optimization?
Groupe Relative Policy Optimization (GRPO) xeetu jàng-jàng buy dooleel la ngir gëna suqali xeetu làkk yi, di àtte tontu bu nekk ci tontub mbokk yu bari ci benn laaj, di dindi reso valeur bu tàqaloo bi PPO di jëfandikoo. Mu siiw ni xeetu tàggat yaram bu mag bi ci ginaaw xeetu xalaat yu DeepSeek.
Ban cër bu mag ci PPO la GRPO dindi?
Coppite siif GRPO mingi wàcce valeur/modèle critique bi PPO jàng, biy nuru benn dayo ak politik bi, di sakkanal memory bu bari ak calcul.
naka la GRPO di xaymaa njariñu benn matal buñ jox?
Njariñu bépp matt mooy (neexal - moyenne groupe) / groupe standard deviation, kon kuréel gi tontu ci benn laaj bi dafay nekk liiñ bu njëkk bi.
Ban xeetu xam-xam moo tax GRPO siiw?
DeepSeek moo dugal GRPO ba noppi siiwal ko, rawatina ci DeepSeekMath ak muy motëru RL bi ci ginaaw xeetu xalaat DeepSeek-R1.
Ban cër la terme KL-divergence def ci GRPO?
Penalty KL ci kaw benn royuwaay (dafay faral di nekk pre-RL) model dafay yamale tàggat yaram suko defee politik bi gëna mëna dem te baña daanu wala dem ci génnug degenerate.
Lan moo waral GRPO di gëna am solo ngir tàggat ay xeetu xalaat ci math wala kode?
Sampling pexe yu bari ak poñ leen ci njubte otomatik saytu ñaar ñu sell ak njariñu GRPO's group-normalised, soxlawul ŋàññi.