Itsinda ryibihembo bisanzwe muri RLHF
Itsinda ryigihembo gisanzwe cyerekana ibihembo byicyitegererezo mugice kimwe cyibisubizo kubibazo bimwe, bihindura amanota yuzuye urusaku mubimenyetso byamahugurwa bihamye.
Incamake
It is the core trick behind GRPO, the algorithm that powers many modern reasoning models.
Kwibira cyane
Mu gushimangira kwigira kubitekerezo byabantu (RLHF), icyitegererezo gitanga ibisubizo kandi icyitegererezo cyigihembo kibatanga amanota, ariko ibihembo mbisi ni urusaku kandi biratandukanye cyane kubitekerezo. Itsinda ryigihembo gisanzwe gikosora ibi muguhitamo itsinda ryibisubizo byinshi kubibazo bimwe, hanyuma bigahindura buri gihembo mugukuramo itsinda risobanura no kugabana gutandukana kwitsinda. Iyi z-amanota iba akarusho. Ubu buryo ni ingenzi kuri Groupe Relative Politiki Optimisation (GRPO), yatangijwe na DeepSeek, yazanaga cyane ibitekerezo bya DeepSeek-R1. Byibanze, GRPO ikuraho umuyoboro wihariye (kunegura) ukoreshwa na PPO, kuva impuzandengo yitsinda ikora nkibanze. Ibi bituma imyitozo yoroshye, ihendutse, kandi yibuka neza mugihe ukomeza ibimenyetso bya gradient neza.
Ubushishozi
Kubwitsinda ryibisubizo hamwe nibihembo r_1 ... r_G, inyungu ni A_i = (r_i - bivuze (r)) / std (r). Ibisubizo byiza kurenza impuzandengo yitsinda ryabo bibona inyungu nziza kandi birashimangirwa; bibi-kurenza-ugereranije basunikwa hasi. Kuberako igereranya rifitanye isano mugihe cyihuse, ibihembo byuzuye byuzuye kandi kuri buri kibazo bigoye guhagarika, kugabanya ibitandukanye. GRPO ituma PPO igabanya intego hamwe na KL ibihano birwanya politiki yo kubuza icyitegererezo kugenda kure cyane.
Ingaruka z'Ingamba
Ibyemezo bisobanutse
Iragufasha gutandukanya ibyifuzo bya tekiniki bisobanutse nururimi rwo kwamamaza.
Igiciro na bije
Urashobora kubaza ibibazo byiza byo gushyira mubikorwa mbere yo gukoresha amafaranga cyangwa igihe.
Itsinda hamwe nakazi
Amakipe asangiye ibitekerezo akora ibicuruzwa byiza, politiki, nibyemezo byo kwiga.
Igihe kizaza cyo guhembera ibihembo bisanzwe muri RLHF
Gushyira hamwe mubisanzwe bitera imbaraga zo gutekereza-moderi itera imbere, aho abanyamideli bigira kubihembo bifatika nkibisubizo nyabyo byimibare nta kunegura wize. Ubushakashatsi burimo kubinonosora: impaka zo gutandukanya gutandukana bisanzwe, gukemura amatsinda yose akosora cyangwa yose yibeshya atanga inyungu zeru, hamwe nubunini bwitsinda. Tegereza amatsinda, kunegura-uburyo bwo gukwirakwiza ibikoresho bya agenti no gukoresha code, aho abagenzuzi bonyine batanga ibimenyetso bihendutse, byinshi byigihembo.
Gushyira mu bikorwa Isi
Guhugura icyitegererezo cyerekana imibare ukoresheje ibisubizo 16 kuri buri kibazo no guhemba abari hejuru yikigereranyo cyukuri.
Kuringaniza neza ubufasha bwa chatbot muguhindura amanota yigihembo-cyitegererezo kubakandida benshi basubiza buri mukoresha.
Kunoza umufasha wa coding aho buri gisubizo cyakemuwe cyatsinzwe niba cyatsinze ibizamini, hanyuma gisanzwe mumatsinda.
Kugabanya ububiko bwa GPU mumuyoboro wa RLHF muguhagarika umuyoboro wa PPO no gukoresha itsinda bivuze nkibanze aho.
Ingaruka & Kurinda
Amakipe atandukanye arashobora gukoresha ijambo rimwe muburyo butandukanye, sobanura intera hakiri kare.
Ibipimo birashobora kugaragara bikomeye mugihe imikorere-yisi-itaringaniye.
Kwirengagiza ubuziranenge bwamakuru na gahunda yo gusuzuma akenshi bitanga ibisubizo byoroshye.
Igishushanyo mbonera
Tangira nururimi rusobanutse rwibisubizo ukeneye.
Toranya intsinzi imwe hamwe nuburyo bumwe bwo gutsindwa mbere yo kwipimisha.
Koresha umuderevu muto hamwe namakuru ahagarariye, ntabwo ari demo yashizweho.
Inyandiko aho Ibihembo Byibisanzwe Bisanzwe muri RLHF bifasha nuburyo bworoshye bworoshye.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Grouped Reward Normalization in RLHF quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Uburebure busanzwe muburyo bwo guhitamo neza
Ibibazo bikunze kubazwa
What is Grouped Reward Normalization in RLHF?
Itsinda ryigihembo gisanzwe cyerekana ibihembo byicyitegererezo mugice kimwe cyibisubizo kubibazo bimwe, bihindura amanota yuzuye urusaku mubimenyetso byamahugurwa bihamye. Nuburiganya bwibanze inyuma ya GRPO, algorithm iha imbaraga moderi nyinshi zo gutekereza.
Mubihembo byitsinda risanzwe, ibihembo bya buri gisubizo ugereranije nibiki?
Buri gihembo gihindurwa kuri z-amanota ukoresheje uburyo busanzwe no gutandukana kwitsinda ryibisubizo kubibazo bimwe.
Niyihe algorithm ifitanye isano cyane no guhembera amatsinda?
GRPO, yatangijwe na DeepSeek kandi ikoreshwa muri DeepSeek-R1, yubatswe muburyo busanzwe bwo guhemba mumatsinda.
Niki kintu kigize PPO gisanzwe GRPO ikuraho?
Ukoresheje impuzandengo yitsinda nkibanze, GRPO ireka abize kunegura, kubika ububiko no kubara.
Bigenda bite kubisubizo ibihembo biri munsi yitsinda ryayo?
Gukuramo itsinda bivuze gukora ibisubizo biri munsi yikigereranyo bifite inyungu mbi, gusunika politiki kure yabo.
Kuki bisanzwe mubisanzwe bigabanya itandukaniro ryamahugurwa?
Kuberako kugereranya bifitanye isano muri buri kibazo, itandukaniro mubipimo byuzuye kandi bigoye gukaraba.