Kwangwa Gutoranya Cyiza-Guhuza
Kwangwa Gutoranya Cyiza-Guhuza (RFT) bitanga ibisubizo byinshi byabakandida, bigakomeza gusa amanota meza, kandi bigakomeza icyitegererezo kubatsinze.
Incamake
It matters because it offers much of RLHF's benefit using straightforward supervised learning instead of complex reinforcement learning.
Kwibira cyane
Kwangwa Gutoranya Cyiza-Guhuza, rimwe na rimwe byitwa ibyiza-bya-N-gutunganya neza, ni ikintu cyingenzi muburyo moderi nka __AIU_PROTECTED_13 __ 'Llama 2 na Llama 3 zahujwe. Ibisobanuro biroroshye: kuri buri gisubizo, tanga ibisubizo byinshi (vuga 4 kugeza 64) uhereye kurugero rwubu, utange amanota buriwese hamwe nigihembo cyigihembo cyangwa umugenzuzi wikora, hanyuma ujugunye ('kwanga') byose usibye ibisubizo biri hejuru. Ibyitegererezo bisigaye byujuje ubuziranenge bihinduka bishya bigenzurwa neza na dataset, kandi icyitegererezo cyabatojwe hamwe nigihombo gisanzwe gikurikira. Gusubiramo uyu muzingo byerekana icyerekezo cyo gutanga ibisubizo byiza wenyine. Kuberako icyitegererezo cyigira kubisubizo byayungurujwe, RFT irinda ihungabana no guhuza imitwe ya politiki-igereranya RL mugihe ikomeje gukoresha ibihembo.
Ubushishozi
RFT ikoresha uburyo bwo gutoranya inshuro nyinshi no kugumya guhembwa igihe kinini ugereranije no gutoranya gukarishye, kurwego rwo hejuru. Amahugurwa kuri abo batsinze binyuze muri cross-entropy isanzwe itandukanya neza iyo myitwarire myiza-ya N igaruka mubyitegererezo by'icyitegererezo kimwe. Kubishobora kugenzurwa nkimibare cyangwa code, 'ibihembo' birashobora gusa kuba igisubizo cyanyuma cyangwa ikizamini cyatsinzwe, bikuraho ibikenewe byigihembo cyize.
Ingaruka z'Ingamba
Umuvuduko n'igipimo
Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
Kugera no kugera
Yagura uburyo bwindimi nuburyo bwo gutumanaho.
Ibyemezo bisobanutse
Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Kazoza ko Kwangwa Gutoranya Cyiza-Guhuza
RFT ni ingenzi cyane mumahugurwa agezweho, akenshi akoreshwa mbere cyangwa kuruhande rwa RL nka PPO na DPO. Ubujurire bwayo bugenda bwiyongera hamwe nibisobanuro bihendutse kandi byikora byikora byikora: uko moderi igenda itera imbere mukubyara no kwisuzumisha, gusubiramo kwangwa byongeye gushyigikira synthique-data hamwe no kwiteza imbere. Tegereza kwishyira hamwe hamwe nuburyo bwo gutekereza butanga urunigi rwukuri rwibitekerezo, hamwe nubushakashatsi bukomeje bwuburyo bwo kwirinda ibihembo byogusenyuka no gutandukana gutandukana mugihe imyitozo inshuro nyinshi kumusaruro wikitegererezo.
Gushyira mu bikorwa Isi
Guhuza imiterere-yuburyo bwa Llama muguhitamo ibisubizo byinshi kuri buri kibazo, kugumana amanota menshi-yerekana amanota, hanyuma SFT kuri ibyo
Gutezimbere imibare ikemura ibisubizo byinshi no kugumana gusa ibigera kubisubizo nyabyo, bisuzumwa
Kode yerekana aho abakandida babikwa gusa iyo batsinze ibizamini byigice, hanyuma bigakoreshwa nkamakuru yamahugurwa
Kubaka inyigisho zifatika zamakuru mugushungura icyitegererezo cyiza ubwacyo cyatanze ibisubizo kumyitozo ikurikira
Ingaruka & Kurinda
Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Igishushanyo mbonera
Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
Komeza kugenzura abantu kugenzura ibisubizo byinshi.
Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Rejection Sampling Fine-Tuning quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
QLoRA na 4-Bit Byiza-Guhuza
Ibibazo bikunze kubazwa
What is Rejection Sampling Fine-Tuning?
Kwangwa Gutoranya Cyiza-Guhuza (RFT) bitanga ibisubizo byinshi byabakandida, bigakomeza gusa amanota meza, kandi bigakomeza icyitegererezo kubatsinze. Nibyingenzi kuko itanga inyungu nyinshi za RLHF ukoresheje imyigire ikurikiranwa neza aho kwiga bigoye gushimangira.
Niki gitekerezo cyibanze cyo Kwangwa Gutoranya Cyiza-Guhuza?
RFT icyitegererezo cyibisubizo byinshi, muyungurura kubatsinze amanota menshi, no guhuza neza icyitegererezo kubatsinze.
Mubishobora kugenzurwa nkimibare cyangwa code, niki gishobora kuba ibihembo?
Kubikorwa bishobora kugenzurwa, RFT irashobora gukoresha neza neza cyangwa gutsinda ibizamini byibice, wirinda ibihembo byizewe.
Nuwuhe muryango wintangarugero uzwi cyane wakoresheje icyitegererezo cyo kwangwa mugihe cyo guhuza?
Meta yasobanuye ukoresheje icyitegererezo cyo kwangwa nkigice cyo gutoza amahugurwa nyuma yicyitegererezo cya Llama 2 na Llama 3.
Ni ukubera iki amakipe ashobora guhitamo RFT kuruta politiki-ya RL nka PPO?
RFT yitoza hamwe nibisanzwe bikurikira-token igihombo ku byitegererezo byayungurujwe, ikuraho ingorane zo guhuza no guhungabana muburyo bwa politiki-buhoro buhoro.
Amahugurwa kuri sample-bihembo byintangarugero akora iki?
Kwiga uhereye hejuru yungurujwe ibisubizo, icyitegererezo cyinjiza imyitwarire yo murwego rwohejuru mugisekuru kimwe.