Normalisation bu neexal buñ boole ci RLHF
Neexal buñ dajale dafay standardisee neexal yi ci benn xeetu tontu ci benn laaj, di soppi poñ yu bari yi ci siñaal tàggat yaram bu dëgër.
Résumé
It is the core trick behind GRPO, the algorithm that powers many modern reasoning models.
Plongeur bu xóot
Ci jàngat buy dooleel ci feedback nit (RLHF), ab xeetu defar ay tontu ak ab xeetu neexal moo leen di jox poñ, waaye neexal yu ñor yi dañu bari xumbaay te dañu wuute lool ci laaj yi. Normalisasioŋ neexal buñ boole dafay saafara lii ci sampling grupu tontu yu bari ci benn laaj, ba noppi normalise neexal bu nekk ci dindi moyenne groupe bi ba noppi xaaj ko ak deviation standard groupe bi. z-score bii mooy nekk njariñ li. Bokk na ci li gëna am solo ci wàllu politik (GRPO), bi DeepSeek dugal, moo gëna dooleel xalaatu DeepSeek-R1. Lu gëna am solo mooy GRPO dindi na reso valeur bu tàqaloo (critique) bi PPO di jëfandikoo, ndax moyenne groupe bi mooy liggéey ci baseline. Loolu dafay yombal tàggat yaram, gëna xéewale, ba noppi gëna sakkanal memory bi, boole ci siñaalu gradient bi nekk ci eskaal bu baax.
Gis-gis xarala
Ngir kuréel gu génne gu am neexal r_1...r_G, njariñ li mooy A_i = (r_i - moyenne(r)) / std(r). Tontu yu gëna baax ci seeni grupu dañuy am njariñ lu baax te dañu leen di dooleel; ñu gëna bon ci moyenne ñu ngi leen di wàcce. Ndax tekkale bi dafa relatif ci diir bu gàtt, eskaalu neexal absolu ak jafe-jafe bu gaaw bu nekk dañu ko fomm, wàññi faraas bi. GRPO dafay tëye mébetu PPO biñ dagg ak daan KL ci wàllu politiku royuwaay ngir moytu model bi dem fu sori.
njeextalu pexe
dogal yu gëna leer
Daf lay jàppale nga tàqale kàddu yu leer ci wàllu xarala ak làkku fësal njaay.
Njëgg ak budget
Mën nga laaj laaj yu gëna baax ci samp gi balaa ngay dugal xaalis wala sa jotu liggéey.
Ekip ak def liggéey
Ekip yi bokk xam-xam ñoo gëna mëna jël yenn dogal ci wàllu produit, politik ak jàng.
Ëlëgu normalisasioŋ bu neexal buñ boole ci RLHF
Normalisasioŋ buñ boole muy gëna dooleel boom modelu xalaat, fu model yi di jàngee ci neexal yiñ mëna saytu lu melni tontu yu jaar yoon ci math te kenn xamul luñu koy ŋàññi. Gëstu yi dañu koy setal: waxtaan ci ndax dañu wara xaaj ci standard deviation, jëflante ak grupu yu baax wala yu baaxul yépp te duñu am benn njariñ, ak yokk dayo grupu bi. Xaarandil pexe yu boole, yu amul benn ŋàññi ngir tasaaroo ci jëfandikoo jumtukaayi agence ak defar kode, fu xoolkati otomatik yi di joxe siñaal yu yomb te bari ay neexal.
Doxal ci àdduna dëgg
Taggat ab xeetu xalaat ci math ci jël misaalu 16 saafara ci jafe-jafe bu nekk ak neexal ñi gëna jub ci grupu bi.
Defar njariñu chatbot ci normalise poñ yi ci xeetu neexal ci tontu yu bari ci kàndidaa yi ci laaj bu jëfandikukat bu nekk.
Yokkateg ab assistant codage fu bépp solution buñu jël ñu jox ko poñ ci ndax jàll na test yu unité yi, ginaaw ga ñu normalise ko ci biir groupe bi.
Wàññi mémoire GPU ci benn pipeline RLHF ci daaneel reso buy ŋàññi PPO te jëfandikoo groupe moyenne bi nekk baseline ci plaas bi.
Risk yi ak balustrade yi
Ekip yu bari mën nañu jëfandikoo benn baat ci anam wu wuute, kon teela leeral yaatuwaayam.
Benchmark yi mën nañu nuru lu am doole waaye performance yi ci àdduna bi duñu tolloo.
Bëgg kalite done ak palaŋu jàngat dafay faral di jur njariñ yu yomba dagg.
Roadmap ngir samp gi
Tàmbaleel ci joxe leeral ci làkk wu leer ci njariñ li nga soxla.
Tannal benn metric bu baax ak benn anam bu baaxul balaa ngay saytu.
Doxal ab pilote bu ndaw ak ay done yu representatif, du ab demo bu leer.
Dokument fu Normalisation Reward Grouped ci RLHF di jàppale ak fu pexe yu gëna yomba gëna baax.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Grouped Reward Normalization in RLHF quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Guddaay normalisasioŋ ci tànneef tànneef
Laaj yi ñuy faral di laaj
What is Grouped Reward Normalization in RLHF?
Neexal buñ dajale dafay standardisee neexal yi ci benn xeetu tontu ci benn laaj, di soppi poñ yu bari yi ci siñaal tàggat yaram bu dëgër. Mooy pexe bi gëna am solo ci ginaaw GRPO, algorithm biy dooleel xeeti xalaat yu bees yu bari.
Ci normalisasioŋ bu neexal buñ boole, ak lan lañuy méngale ak neexal bu tontu bu nekk?
Neexal bu nekk dañu koy soppi def ko z-score ci jëfandikoo moyenne bi ak standard deviation bu grupu tontu yi ci benn laaj bi.
Ban algorithm moo gëna méngoo ak normalisasioŋ bu neexal buñ boole?
GRPO, DeepSeek moo ko dugal ba noppi jëfandikoo ko ci DeepSeek-R1, dafa tabax ci normalise neexal yi ci biir benn kuréel.
Ban cër ci PPO standard la GRPO di dindi?
Suñu jëfandikoo moyenne groupe bi muy ligne de base, GRPO dafay dindi kritik bi jàng, sakkanal mémoire ak calcul.
Lan mooy dall tontu bu neexal bi nekk ci suufu li grupu bi mëna am?
Dindi moyenne groupe bi dafay tax tontu yu nekk ci suufu moyenne am njariñ bu baaxul, di push politik bi fu sori ci ñoom.
Lan moo waral normalisasioŋ ci biir benn kuréel wàññi faraasu tàggat yaram?
Ndax tekkale yi dañu relatif ci biir prompt bu nekk, wuute ci escale absolu ak jafe-jafe prompt raxas.