Inganta Manufofin Dangi na Ƙungiya
Haɓaka Manufofin Dangantaka na Ƙungiya (GRPO) hanya ce ta ƙarfafawa-koyo don ingantaccen tsarin harshe wanda ke yin hukunci akan kowace amsa a kan gungun 'yan'uwan amsoshi guda ɗaya, kawar da keɓantaccen hanyar sadarwar ƙima da PPO ke amfani da shi.
Dubawa
It became famous as the core training trick behind DeepSeek's reasoning models.
Zurfafa nutsewa
GRPO bambance-bambancen koyarwa-gradient ƙarfafa koyo da aka ƙera don sanya RL kyakkyawan daidaita manyan nau'ikan harshe mai rahusa da kwanciyar hankali. Madaidaicin PPO yana buƙatar ƙwararren 'mai suka' (samfurin ƙima), gwargwadon girman manufofin kanta, don kimanta yadda kowane alamar ke da kyau. GRPO yana cire wannan suka gaba ɗaya. Ga kowane faɗakarwa yana ba da samfuran ƙungiyar gamawa (a ce 8-64), ya ƙididdige su duka tare da siginar lada, sannan ya ƙididdige fa'idar kowane kammalawa ta hanyar daidaita ladan sa a kan ma'anar ƙungiyar da madaidaicin karkata. Amsoshi sama da matsakaici ana ƙarfafa su kuma waɗanda ke ƙasa da matsakaicin an danne su. Kalmar bambance-bambancen KL tana kiyaye ƙirar kusa da manufar tunani. DeepSeek ya gabatar da shi, yana ƙarfafa DeepSeekMath da ƙirar tunani DeepSeek-R1.
Fahimtar Fasaha
Babban ra'ayin shine maye gurbin tushen ƙimar koyo na PPO tare da tushen rukunin Monte Carlo. Ga rukunin abubuwan da aka fitar tare da lada r_i, kowane fa'ida shine A_i = (r_i - ma'ana (r)) / std (r). Wannan ƙimar da aka daidaita tana haɓaka ƙimar yuwuwar da aka yanke, daidai kamar yadda yake a cikin PPO, da kuma hukuncin KL akan daskararrun ƙirar ƙira yana hana drift. Saboda babu wani mai sukar da aka horar, ƙwaƙwalwar ajiya da ƙididdige kusan rabin, kuma daidaitawa da sauri yana ba da ma'auni, ƙananan fa'idodi.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Inganta Manufofin Dangantaka na Ƙungiya
GRPO da sauri ya zama tsohuwar girke-girke don horar da ƙirar tunani mai buɗewa, kuma labs suna ta maimaitawa akan raunin sa. Masu bincike suna binciko gyare-gyare don tsayi da wahala (kamar Dr. GRPO), matakin-alama maimakon daidaita matakin-jeri, da cirewa ko sake fasalin kalmar KL. Yi tsammanin haɗin kai tare da tabbataccen lada (lissafi, lamba, amfani da kayan aiki), mafi kyawun sarrafa sigina marasa ƙarfi, da matasan da ke haɗa tushen rukuni tare da masu sukar nauyi don wakili, ayyuka masu yawa.
Aiwatar da Gaskiyar Duniya
Horar da DeepSeek-R1 da DeepSeekMath don samar da dogon jerin tunani na tunani ta amfani da lada na daidaitattun ƙa'ida akan matsalolin lissafi.
Kyawawan ƙirar ƙira-ƙira mai kyau inda kowane samfurin da aka ƙirƙira ya sami maki ta ko ya wuce gwaje-gwaje na yanki, kuma an daidaita ƙungiyar don zaɓar masu nasara.
Buɗaɗɗen bututun RLHF (misali, a cikin ɗakunan karatu na TRL da verl) ta amfani da GRPO don daidaita samfuran taɗi ba tare da biyan kuɗin hanyar sadarwa ta daban ba.
Haɓaka bin umarni ko ɗabi'a na aminci ta hanyar samar da amsoshi da yawa a cikin gaggawa da kuma ba da lada ga waɗanda ƙimar samfurin lada mafi girma dangane da takwarorinsu.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Group Relative Policy Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Ingantaccen Manufofin Kusa
Tambayoyin da ake yawan yi
What is Group Relative Policy Optimization?
Haɓaka Manufofin Dangantaka na Ƙungiya (GRPO) hanya ce ta ƙarfafawa-koyo don ingantaccen tsarin harshe wanda ke yin hukunci akan kowace amsa a kan gungun 'yan'uwan amsoshi guda ɗaya, kawar da keɓantaccen hanyar sadarwar ƙima da PPO ke amfani da shi. Ya zama sananne azaman babban dabarar horarwa a bayan ƙirar tunanin DeepSeek.
Wane babban sashi na PPO GRPO ya kawar?
Canjin sa hannu na GRPO yana sauke ƙimar koyo/samfurin sukar PPO, wanda yawanci kusan girman daidai yake da manufofin, yana adana babban ƙwaƙwalwar ajiya da ƙididdigewa.
Ta yaya GRPO ke lissafta fa'idar don kammalawa?
Kowane fa'idar gamawa shine (lada - ma'anar rukuni) / daidaitattun daidaiton rukuni, don haka rukunin amsoshi iri ɗaya yana aiki kamar tushe.
Wadanne samfura ne suka sanya GRPO sananne?
DeepSeek ya gabatar da GRPO kuma ya shahara, musamman a cikin DeepSeekMath kuma a matsayin injin RL a bayan ƙirar DeepSeek-R1.
Wace rawa kalmar bambance-bambancen KL ke takawa a cikin GRPO?
Hukuncin KL akan wani tunani (yawanci pre-RL) ƙirar tana daidaita horo don haka manufar ta inganta ba tare da rugujewa ko ɓata ba.
Me yasa GRPO ke da kyau musamman ga tsarin horar da tunani akan lissafi ko lamba?
Samfuran mafita da yawa da ƙima su tare da daidaitattun daidaitattun nau'i-nau'i da tsafta tare da fa'idodin ƙungiyar GRPO, babu mai sukar da ake buƙata.