Caadiyan Abaalmarinta Kooxaysan ee RLHF
Caadi ahaan abaal-marin-kooxeedku waxa ay jaangooyaa abaal-marinaha moodeelka gudahooda dufcad jawaabaha isla isla markiiba, u beddeleysa buundooyinka buuqa calaamad tababar deggan.
Dulmar
It is the core trick behind GRPO, the algorithm that powers many modern reasoning models.
quusid qoto dheer
Marka la xoojinayo barashada jawaab celinta aadanaha (RLHF), moodalku wuxuu soo saaraa jawaabo wuxuuna moodel abaalgud u keenaa iyaga, laakiin abaal-marinaha cayriin waa qaylo waxayna ku kala duwan yihiin dardargelinta. Caadi ahaan abaal-marin-kooxeedku waxa ay hagaajinaysaa tan iyada oo la tijaabinayo koox dhawr jawaabo ah isla isla markiiba, ka dibna caadi ka dhigaysa abaal-marin kasta iyada oo la jarayo macnaha kooxda oo loo qaybinayo weecis heerka kooxda ah. Dhibcaha z-kan ayaa noqonaya faa'iidada. Habkani waxa uu udub dhexaad u yahay Horumarinta Siyaasadda Qaraabada ee Kooxda (GRPO), oo ay soo bandhigtay DeepSeek, kaas oo si caan ah u xoojiya fikradda DeepSeek-R1. Muhiimad ahaan, GRPO waxay meesha ka saaraysaa isku-xidhka qiimaha gaarka ah (naqdiye) ee ay isticmaasho PPO, maadaama celceliska kooxuhu u adeegaan sida gundhigga. Tani waxay tababarka ka dhigaysa mid fudud, ka raqiisan, oo wax ku ool u leh xusuusta iyadoo la ilaalinayo calaamada gradient-ka mid miisaan leh.
Aragtida Farsamada
Koox wax soo saar leh oo leh abaal-marin r_1...r_G, faa'iidada waa A_i = (r_i − macnaheedu(r)) / std(r). Jawaabaha ka fiican celceliska kooxdooda waxay helayaan faa'iido wanaagsan waana la xoojiyaa; kuwa ka liita celceliska ayaa hoos loo riixaa. Sababtoo ah isbarbardhigga waa qaraabo isla markiiba, qiyaasta abaal-marin buuxda iyo dhib degdeg ah oo la joojiyo, yaraynta kala duwanaanshaha. GRPO waxa ay ilaalisaa ujeedada la gooyey ee PPO iyo ganaaxa KL ee ka dhanka ah siyaasada tixraaca si looga hortago in qaabka uu aad u fogaado.
Saamaynta Istiraatijiyadeed
Go'aamo cad
Waxay kaa caawinaysaa inaad kala saartid sheegashooyinka farsamada cad iyo luqadda suuq-geynta.
Qiimaha iyo miisaaniyada
Waxaad waydiin kartaa su'aalo fulineed oo wanaagsan ka hor inta aadan lacag ama waqti bixin.
Kooxda iyo socodka shaqada
Kooxaha fahamka la wadaago waxay sameeyaan wax soo saar, siyaasad, iyo go'aano waxbarasho oo wanaagsan.
Mustaqbalka Caadiyan Abaalmarinta Kooxaysan ee RLHF
Caadi ahaan kooxaysan ayaa sii hurinaysa kobaca moodeelka, halkaas oo moodellada wax ka bartaan abaal-marinnada la xaqiijin karo sida jawaabaha xisaabta saxda ah iyada oo aan la helin dhaleecayn la bartay. Cilmi-baadhistu waxay sifaynaysaa: doodaha ku saabsan in lagu qaybsado beddelka caadiga ah, la tacaalida kooxaha sax ah ama dhammaan khaladka ah ee soo saara faa'iidada eber, iyo cabbirida tirada kooxda. Filo qaabab kooxaysan, dhaleecayn la'aan si loogu faafiyo isticmaalka aaladaha wakiilka ah iyo soo saarista koodka, halkaas oo xaqiijiyaasha tooska ah ay bixiyaan calaamado abaal-marin ah oo jaban.
Dhaqangelinta Adduunka-dhabta ah
Tababarka qaabka xisaabinta-macnaha iyadoo la qaadayo 16 xal oo dhib kasta ah oo lagu abaalmariyo kuwa ka sarreeya celceliska saxda ah ee kooxda.
Hagaajinta waxtarka chatbot-ka iyada oo caadi ka dhigaysa buundooyinka moodeelka abaal-marinaha ee musharraxiinta dhowr ah ayaa jawaabaha isticmaalaha kasta.
Hagaajinta kaaliyaha codaynta halkaas oo xal kasta oo muunad ah lagu qiimeeyay haddii ay ku gudubto imtixaannada cutubka, ka dibna caadi laga dhigo kooxda dhexdeeda.
Yaraynta xusuusta GPU ee dhuumaha RLHF iyada oo la tuurayo shabakadda dhaleeceynta PPO iyo adeegsiga kooxda macnaheedu waa saldhigga.
Khatarta & Dariiqyada Ilaalada
Kooxo kala duwan ayaa laga yaabaa inay isla erey u isticmaalaan si kala duwan, marka hore u qeex baaxadda.
Tilmaamaha ayaa u ekaan kara kuwo xooggan halka waxqabadka dhabta ah ee dunidu aanu sinnayn.
In la iska indho tiro tayada xogta iyo qorshayaasha qiimayntu waxay inta badan abuurtaa natiijooyin jilicsan.
Qorshe Hawleedka Dhaqangelinta
Ka bilow qeexidda luqadda cad ee natiijada aad u baahan tahay.
Dooro hal cabbir guusha iyo hal xaalad guuldarro ka hor tijaabada.
Ku orod duuliye yar oo wata xogta matale, ee ma aha bandhig muuqaal ah.
Dukumeenti halka Abaalmarinta Kooxaysan ee RLHF ay caawiso iyo meelaha hababka fudud ay ka fiican yihiin.
Sii wad Sahaminta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Grouped Reward Normalization in RLHF quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Hagaha xiga
Caadiyan dhererka ee hagaajinta doorbidida
Su'aalaha soo noqnoqda
What is Grouped Reward Normalization in RLHF?
Caadi ahaan abaal-marin-kooxeedku waxa ay jaangooyaa abaal-marinaha moodeelka gudahooda dufcad jawaabaha isla isla markiiba, u beddeleysa buundooyinka buuqa calaamad tababar deggan. Waa khiyaamada xudunta u ah GRPO, algoorithm-ka awood u leh noocyo badan oo caqli-gal ah oo casri ah.
Abaalmarinta kooxaysan ee caadiga ah, waa maxay abaalgudka jawaab kasta marka la barbar dhigo?
Abaal marin kasta waxaa loo rogaa z-dhibcaha iyadoo la adeegsanayo celceliska iyo leexashada caadiga ah ee kooxda jawaabaha isla markiiba.
Algorithmmeekee ayaa inta badan la xidhiidha caadiyaynta abaalgudka kooxaysan?
GRPO, oo ay soo bandhigtay DeepSeek oo lagu isticmaalo DeepSeek-R1, ayaa lagu dhisay hab caadi ah oo lagu abaalmariyo kooxda dhexdeeda.
Waa maxay qayb ka mid ah PPO-ga caadiga ah ee GRPO si gaar ah u baabi'iyo?
Adigoo isticmaalaya celceliska kooxda sida gundhigga, GRPO waxay hoos u dhigtaa dhaleeceynta la bartay, keydinta xusuusta iyo xisaabinta.
Maxaa ku dhacaya jawaabta abaalgudkeedu ka hooseeyo macnaha kooxdeeda?
Ka-goynta kooxda macnaheedu waxay ka dhigaysaa jawaabaha celceliska hooseeya waxay leeyihiin faa'iidooyin taban, oo ka riixaya siyaasadda iyaga.
Waa maxay sababta caadi ka dhigista kooxdu ay u dhimayso kala duwanaanshaha tababarka?
Sababtoo ah isbarbardhiggu waa mid udhaxeeya isla markiiba, kala duwanaanshaha miisaanka saxda ah iyo dhibka degdega ah ee dhaqidda.