Ƙarfafa Koyo Daga Sauraron Dan Adam
RLHF ita ce dabarar da ke juya samfurin ɗanyen harshe zuwa mataimaki mai taimako, mai ladabi ta horar da shi akan abubuwan ɗan adam.
Dubawa
It matters because it aligns model behavior with what people actually want, not just what is statistically likely.
Zurfafa nutsewa
Samfurin harshe da aka riga aka horar yana tsinkayar rubutu mai ma'ana, amma tabbataccen ba daidai yake da taimako, gaskiya ko aminci ba. RLHF yana gyara wannan a matakai. Na farko, gyare-gyaren da ake kulawa yana koya wa ƙirar bin umarni ta amfani da amsoshi da aka rubuta na ɗan adam. Bayan haka, mutane suna kwatanta nau'i-nau'i na amsoshi samfurin zuwa ga sauri ɗaya kuma su zaɓi mafi kyau; waɗannan kwatancen suna horar da nau'in lada daban wanda ke nuna kowane amsa. A ƙarshe, ƙirar harshe an inganta shi tare da ƙarfafa koyo don samar da martani da ƙimar samfurin lada sosai. Hukunci yana hana shi yin nisa sosai daga ƙirar asali don haka ya kasance mai kyau kuma baya amfani da ƙima na samfurin lada. RLHF ita ce tsakiyar yin amfani da mataimakan salon ChatGPT.
Fahimtar Fasaha
Samfurin lada galibi ana horar da su akan nau'i-nau'i na zaɓi tare da asarar salon Bradley-Terry, koyo don ba da amsar da aka fi so da ɗan adam maki mafi girma. Sannan ana sabunta manufofin tare da PPO (Proximal Policy Optimization), wanda ke haɓaka lada yayin da hukuncin rarrabuwar kawuna na KL akan tsarin tunani yana hana haɓakawa fiye da kima da 'Hacking ɗin lada'. Saboda PPO yana da aminci, sababbin hanyoyin kamar DPO (Maganin fifikon fifikon kai tsaye) suna tsallake ƙirar lada na zahiri da madauki na ƙarfafawa, inganta manufofin kai tsaye daga nau'ikan zaɓin zaɓi.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Ƙarfafa Koyo Daga Sauraron Dan Adam
Ana daidaita RLHF kuma ana sarrafa shi ta wani bangare. DPO da hanyoyin fifita kai tsaye masu alaƙa suna maye gurbin bututun PPO mai nauyi don ƙungiyoyi da yawa, kuma RLAIF yana amfani da ra'ayoyin AI da aka ƙirƙira (kamar yadda yake cikin Tsarin Tsarin Mulki) don rage farashin alamar. Bincike yana magance satar lada, son zuciya, da wahalar yanke hukunci mai tsayi ko ƙwararrun martani, tare da dabaru kamar sa ido kan tsari da muhawara. Yi tsammanin daidaitawa don haɗa ra'ayoyin ɗan adam da AI, mafi kyawun sigina na lada fiye da babban yatsa guda ɗaya, da haɓaka binciken wanda ke ba da zaɓin da waɗanne ƙimar da suka ƙididdigewa.
Aiwatar da Gaskiyar Duniya
Tuna mataimakan taɗi don ya ƙi buƙatun cutarwa kuma yana ba da amsoshi ingantattun ingantattun amsoshi maimakon kawai rubutu mai ma'ana.
Matsayin taƙaitawa nau'i-nau'i ta zaɓin ɗan adam don horar da ƙirar da ke rubuta taƙaitattun mutane a zahiri suna da amfani.
Rage sakamako mai guba ko son zuciya ta hanyar amsa mai lada wanda masu kimar ɗan adam ke yanke hukunci mai mutuntawa da aminci.
Amfani da DPO akan bayanan da aka fi so vs. amsoshi da aka ƙi don daidaita ƙirar tushen buɗe ido ba tare da gudanar da cikakken madauki na PPO ba.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Reinforcement Learning From Human Feedback quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Ƙarfafa Koyo
Tambayoyin da ake yawan yi
What is Reinforcement Learning From Human Feedback?
RLHF ita ce dabarar da ke juya samfurin ɗanyen harshe zuwa mataimaki mai taimako, mai ladabi ta horar da shi akan abubuwan ɗan adam. Yana da mahimmanci saboda yana daidaita halayen ƙirar da abin da mutane ke so a zahiri, ba kawai abin da ke da yuwuwar ƙididdiga ba.
Menene babban manufar RLHF don samfurin harshe?
RLHF yana jagorantar abin ƙira ga martanin da mutane suka fi so, yana mai da shi ƙarin taimako, gaskiya, da aminci maimakon kawai a iya ganewa.
Menene samfurin lada a cikin RLHF a zahiri ya koya yi?
An horar da ƙirar tukwici akan kwatancen fifikon ɗan adam don samun sakamako mai ƙima, yana ba da siginar da manufar ta inganta da.
Me yasa ake amfani da hukuncin bambance-bambancen KL akan ainihin samfurin da aka yi amfani da shi yayin matakin RL?
Hukuncin KL yana kiyaye ingantattun manufofin kusa da ƙirar ƙira, da kiyaye haƙƙin satar lada da asarar ƙwarewa.
Ta yaya ake tattara bayanan zaɓi don ƙirar lada?
Masu ba da labari suna zaɓar amsar da aka fi so a cikin kwatance biyu, waɗanda ke horar da ƙirar lada ta hanyar asarar salon Bradley-Terry.
Wace fa'ida DPO (Haɓaka fifiko kai tsaye) ke bayarwa akan bututun RLHF na gargajiya?
DPO yana samun makasudin kai tsaye daga abubuwan da aka zaɓa, yana guje wa keɓantaccen samfurin lada da matakin ƙarfafawa mai aminci.