Nhazi nhazi ụgwọ ọrụ agbakọtara na RLHF
Ngwọta ugwo agbakọkọtara ọnụ na-ahazi ụgwọ ọrụ ihe nlereanya n'ime ọnụọgụ nzaghachi maka otu ngwa ngwa, na-atụgharị akara mkpọtụ ka ọ bụrụ mgbama ọzụzụ kwụsiri ike.
Nchịkọta
It is the core trick behind GRPO, the algorithm that powers many modern reasoning models.
Ime miri emi
N'ịmụta nkwado sitere na nzaghachi mmadụ (RLHF), ihe nlereanya na-ewepụta nzaghachi yana ụdị ụgwọ ọrụ na-enye ha ọnụ, mana ụgwọ ọrụ nkịtị na-eme mkpọtụ ma na-adịgasị iche n'ofe mkpali. Nkwụghachi ụgwọ ọrụ agbakọtara na-edozi nke a site n'ịtụle otu nzaghachi dị iche iche maka otu ngwa ngwa, wee mezie ụgwọ ọrụ nke ọ bụla site n'iwepụ ihe pụtara otu na kewaa site n'usoro ọkọlọtọ otu. Nke a z-akara na-aghọ uru. Ụzọ a bụ isi na njikarịcha amụma amụma otu (GRPO), nke DeepSeek webatara, bụ nke kwadoro echiche DeepSeek-R1 nke ama ama. N'ụzọ dị mkpa, GRPO na-ewepụ netwọkụ bara uru dị iche iche (onye nkatọ) nke PPO na-eji, ebe ọ bụ na nkezi otu na-arụ ọrụ dị ka ntọala. Nke a na-eme ka ọzụzụ dị mfe, dị ọnụ ala, na ebe nchekwa karịa ka ị na-edobe mgbama gradient nke ọma.
Nghọta nka nka
Maka otu ihe nrụpụta nwere ụgwọ ọrụ r_1...r_G, uru ya bụ A_i = (r_i - mean(r)) / std(r). Azịza dị mma karịa nkezi otu ha na-enweta uru dị mma ma na-ewusi ya ike; ndị dị njọ karịa nkezi ka a na-akwatu ala. N'ihi na ntụnyere dị n'ime ngwa ngwa, ụgwọ ọrụ zuru oke yana ihe isi ike ozugbo ịkagbu, na-ebelata ọdịiche. GRPO na-edobe ebumnobi PPO gbubiri yana ntaramahụhụ KL megide amụma ntụaka iji gbochie ihe nlereanya ịfefe nke ukwuu.
Mmetụta atụmatụ
Mkpebi doro anya
Ọ na-enyere gị aka ikewapụta nkwupụta ọrụ aka doro anya na asụsụ ahịa.
Ọnụ ego na mmefu ego
Ị nwere ike ịjụ ajụjụ mmejuputa iwu ka mma tupu itinye ego ma ọ bụ oge.
Team na usoro ọrụ
Ndị otu nwere nghọta na-eme ka ngwaahịa, amụma na mkpebi mmụta ka mma.
Ọdịnihu nke nhazi ụgwọ ọrụ agbakọtara na RLHF
Nhazi nhazi nke agbakọta na-eme ka echiche echiche na-aga n'ihu, ebe ụdị na-amụta site na ụgwọ ọrụ enwere ike ime dị ka azịza mgbakọ na mwepụ ziri ezi na-enweghị onye nkatọ mụtara. Nnyocha na-emezigharị ya: arụmụka maka ma a ga-ekewa site na ọkọlọtọ ọkọlọtọ, ijikwa otu ndị ziri ezi ma ọ bụ ndị na-ezighị ezi na-emepụta uru efu, na ịmepụta nha otu. Na-atụ anya ụzọ agbakọtara ọnụ, ụzọ enweghị nkatọ ka a ga-agbasa na iji ngwa ngwa na ọgbọ koodu, ebe ndị na-enyocha akpaaka na-enye akara ụgwọ ọrụ dị ọnụ ala.
Mmejuputa n'ezie n'ụwa
Ịzụ ihe nlereanya nke mgbakọ na mwepụ site n'ịtụle ngwọta 16 maka nsogbu ọ bụla na ịkwụghachi ndị ahụ karịa nkezi ziri ezi nke otu ahụ.
Idozi enyemaka chatbot nke ọma site n'ịhazigharị akara nrịbama-ụgwọ ọrụ gafee ọtụtụ ndị ndoro-ndoro na-aza onye ọrụ ọ bụla ozugbo.
Imelite onye inyeaka koodu ebe a na-enweta azịza nlele ọ bụla site na ma ọ gafere ule otu, wee mee ka ọ dị mma n'ime otu.
Ibelata ebe nchekwa GPU na pipeline RLHF site na idobe netwọkụ nkatọ PPO na iji otu pụtara dị ka ntọala kama.
Ihe ize ndụ & okporo ụzọ nche
Otu dị iche iche nwere ike iji otu okwu ahụ mee ihe n'ụzọ dị iche, yabụ kọwapụta oge n'oge.
Ihe nrịbama nwere ike ịdị ike ebe arụmọrụ ụwa na-adaghị adaba.
Ileghara ogo data na atụmatụ nyocha anya na-emepụtakarị nsonaazụ na-adịghị mma.
Map mmejuputa
Malite na nkọwa asụsụ dị larịị nke nsonaazụ ịchọrọ.
Họrọ otu metrik ịga nke ọma na otu ọnọdụ ọdịda tupu nnwale.
Gbaa obere onye na-anya ụgbọ elu nwere data nnọchite anya, ọ bụghị ihe ngosi ngosi na-egbu maramara.
Detuo ebe nhazi ọkwa ụgwọ ọrụ agbakọtara na RLHF na-enyere aka yana ebe ụzọ dị mfe ka mma.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Grouped Reward Normalization in RLHF quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Ndozi ogologo na nkwalite mmasị
Ajụjụ a na-ajụkarị
What is Grouped Reward Normalization in RLHF?
Ngwọta ugwo agbakọkọtara ọnụ na-ahazi ụgwọ ọrụ ihe nlereanya n'ime ọnụọgụ nzaghachi maka otu ngwa ngwa, na-atụgharị akara mkpọtụ ka ọ bụrụ mgbama ọzụzụ kwụsiri ike. Ọ bụ aghụghọ dị n'azụ GRPO, algọridim na-akwado ọtụtụ ụdị echiche ọgbara ọhụrụ.
N'ime nhazigharị ugwo dị n'ìgwè, gịnị bụ ụgwọ ọrụ nzaghachi ọ bụla tụnyere?
A na-atụgharị ụgwọ ọrụ ọ bụla ka ọ bụrụ akara z-score site na iji ngbanwe pụtara na ọkọlọtọ nke otu nzaghachi na otu ngwa ngwa.
Kedu algọridim kacha ejikọta ya na nhazi ụgwọ ọrụ agbakọtara ọnụ?
GRPO, nke DeepSeek webatara ma jiri ya mee ihe na DeepSeek-R1, bụ nke ewuru ya gburugburu ịhazigharị ụgwọ ọrụ n'ime otu.
Kedu akụkụ nke ọkọlọtọ PPO ka GRPO na-ewepụ nke ọma?
Site n'iji nkezi otu dị ka ntọala, GRPO tụda onye nkatọ mmụta, na-echekwa ebe nchekwa na gbakọọ.
Kedu ihe ga-eme nzaghachi nke ụgwọ ọrụ ya dị n'okpuru ọnụ ahịa otu ya?
Mwepụ otu pụtara na-eme ka nzaghachi ndị dị n'okpuru na-enwe uru na-adịghị mma, na-akwapụ iwu ahụ n'ebe ha nọ.
Kedu ihe kpatara ime ka ọ dị mma n'ime otu na-ebelata ọdịiche ọzụzụ?
N'ihi na ntụnyere dị n'ime ngwa ngwa ọ bụla, ndịiche dị n'ogo zuru oke yana ihe isi ike na-asachapụ.