Daidaita Ladan Rukuni a cikin RLHF
Daidaita ladan da aka haɗe-haɗe yana daidaita ladan ƙira a cikin juzu'in martani ga gaugawa iri ɗaya, yana mai da ƙima mai hayaniya zuwa siginar horarwa.
Dubawa
It is the core trick behind GRPO, the algorithm that powers many modern reasoning models.
Zurfafa nutsewa
A cikin ƙarfafa koyo daga ra'ayoyin ɗan adam (RLHF), samfurin yana haifar da martani kuma samfurin lada ya ƙididdige su, amma ɗanyen lada suna hayaniya kuma sun bambanta sosai a cikin faɗakarwa. Matsakaicin lada mai rukuni yana gyara wannan ta hanyar samar da gungun martani da yawa zuwa ga sauri daya, sannan daidaita kowane tukuicin ta hanyar rage ma'anar kungiyar da kuma rarraba ta daidaitattun daidaiton kungiyar. Wannan z-score ya zama fa'ida. Hanyar ita ce tsakiya ga Haɓaka Manufofin Dangantaka na Ƙungiya (GRPO), wanda DeepSeek ya gabatar, wanda ya shahara wajen ƙarfafa tunanin DeepSeek-R1. Mahimmanci, GRPO yana kawar da keɓantaccen hanyar sadarwa na ƙima (mai sukar) wanda PPO ke amfani dashi, tunda matsakaicin rukuni yana aiki azaman tushen tushe. Wannan yana sa horo ya zama mai sauƙi, mai rahusa, kuma mafi ingantaccen ƙwaƙwalwar ajiya yayin kiyaye siginar gradient da kyau.
Fahimtar Fasaha
Ga rukunin abubuwan da aka fitar tare da lada r_1...r_G, fa'idar ita ce A_i = (r_i - ma'ana (r)) / std (r). Amsoshi mafi kyau fiye da matsakaicin ƙungiyar su suna samun fa'ida mai kyau kuma ana ƙarfafa su; mafi muni fiye da matsakaici ana tura su ƙasa. Saboda kwatancen dangi ne a cikin gaggawa, cikakkiyar ma'aunin lada da wahala mai sauri sokewa, rage bambance-bambance. GRPO yana kiyaye manufar PPO da yanke hukuncin KL a kan manufar tunani don hana samfurin yin nisa sosai.
Dabarun Tasiri
Shawarwari masu haske
Yana taimaka muku keɓance bayyanannen da'awar fasaha daga harshen talla.
Kudin da kasafin kuɗi
Kuna iya yin mafi kyawun tambayoyin aiwatarwa kafin kashe kuɗi ko lokaci.
Ƙungiya da aikin aiki
Ƙungiyoyin da ke da fahimtar juna suna yin mafi kyawun samfura, manufofi, da yanke shawara na koyo.
Makomar Daidaita Ladan Rukuni a cikin RLHF
Haɗin kai yana haifar da haɓakar tunani-samfurin, inda samfura ke koyo daga tabbataccen lada kamar amsoshin lissafi daidai ba tare da ƙwararren mai suka ba. Bincike yana tace shi: muhawara kan ko za a raba ta hanyar daidaitattun daidaito, sarrafa daidaitattun kungiyoyi ko duk kuskuren da ke haifar da fa'ida, da daidaita girman rukuni. Yi tsammanin haɗe-haɗe, hanyoyin da ba su da zargi don yadawa zuwa amfani da kayan aiki na kayan aiki da ƙirƙira lambar, inda masu tabbatarwa ta atomatik ke ba da siginar arha, ɗimbin lada.
Aiwatar da Gaskiyar Duniya
Horar da ƙirar lissafi ta hanyar samar da mafita guda 16 a kowace matsala da kuma ba da lada ga waɗanda ke sama da matsakaicin daidaiton ƙungiyar.
Daidaita taimakon chatbot ta hanyar daidaita ƙima-samfurin lada a cikin yawancin amsawar ɗan takara ga kowane mai amfani da sauri.
Haɓaka mataimaki na coding inda kowane samfurin bayani ya sami maki ta ko ya wuce gwaje-gwaje na raka'a, sannan an daidaita shi a cikin rukuni.
Rage ƙwaƙwalwar GPU a cikin bututun RLHF ta hanyar watsar da hanyar sadarwa ta masu sukar PPO da amfani da ƙungiyar ma'anar azaman tushe maimakon.
Hatsari & Tsare-tsare
Ƙungiyoyi daban-daban na iya amfani da kalmar iri ɗaya daban, don haka ayyana iyaka da wuri.
Alamomi na iya yin kama da ƙarfi yayin da aikin zahirin duniya bai yi daidai ba.
Yin watsi da ingancin bayanai da tsare-tsaren kimantawa galibi yana haifar da sakamako mara ƙarfi.
Taswirar Hanya
Fara da ma'anar harshe a sarari na sakamakon da kuke buƙata.
Zaɓi ma'aunin nasara ɗaya da yanayin gazawa ɗaya kafin gwaji.
Gudun ƙaramin matukin jirgi tare da bayanan wakilci, ba saitin demo da aka goge ba.
Daftarin aiki inda Rukuni na Daidaita Lada a cikin RLHF yana taimakawa kuma inda mafi sauƙi hanyoyin suka fi kyau.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Grouped Reward Normalization in RLHF quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Tsawon Tsawon Matsakaici a cikin Inganta fifiko
Tambayoyin da ake yawan yi
What is Grouped Reward Normalization in RLHF?
Daidaita ladan da aka haɗe-haɗe yana daidaita ladan ƙira a cikin juzu'in martani ga gaugawa iri ɗaya, yana mai da ƙima mai hayaniya zuwa siginar horarwa. Ita ce babbar dabarar da ke bayan GRPO, algorithm ɗin da ke ba da iko da yawancin ƙirar tunani na zamani.
A cikin daidaitawar lada mai rukuni, menene ladan kowane amsa idan aka kwatanta?
Ana canza kowane lada zuwa makin z-score ta amfani da ma'ana da madaidaicin karkata na rukunin martani ga saƙo ɗaya.
Wanne algorithm ne ya fi alaƙa da daidaitawar lada?
GRPO, wanda DeepSeek ya gabatar kuma aka yi amfani da shi a DeepSeek-R1, an gina shi ne ta hanyar daidaita lada a cikin rukuni.
Wane bangare na daidaitaccen PPO GRPO ke kawar da shi musamman?
Ta amfani da matsakaita na rukuni azaman tushen tushe, GRPO yana sauke mai sukar da aka koya, adana ƙwaƙwalwar ajiya da ƙididdigewa.
Me zai faru da martani wanda ladansa bai kai ma'anar kungiyarsa ba?
Rage ma'anar ƙungiyar yana sa martanin ƙasa-matsakaici yana da fa'ida mara kyau, yana kawar da manufofin daga gare su.
Me yasa daidaitawa a cikin ƙungiya yana rage bambancin horo?
Saboda kwatancen dangi ne a cikin kowane faɗakarwa, bambance-bambance a cikin cikakkiyar ma'auni da saurin wahala.