Uburebure busanzwe muburyo bwo guhitamo neza
Uburebure busanzwe buhindura intego-yoguhuza intego kugirango moderi ihagarike gutsindira kwemererwa kwandika ibisubizo birebire.
Incamake
It matters because uncorrected reward signals push chatbots toward verbose, padded responses instead of genuinely better ones.
Kwibira cyane
Iyo moderi ihujwe nuburyo nka RLHF cyangwa DPO, bigira kubigereranya aho abantu (cyangwa icyitegererezo cyigihembo) bahisemo 'ibyiza' byibisubizo bibiri. Ikosa rikomeje ni uko ibisubizo birebire bikunda gukundwa nubwo bitaba byiza mubyukuri, icyitegererezo rero cyiga shortcut: vuga ijambo. Uburebure busanzwe burwanya ibi. Muri DPO ibihembo bidasubirwaho ni igiteranyo cya buri token log-probability itandukanye, ikura muburyo bwuburebure. Ibihinduka nkuburebure-busanzwe DPO na SimPO bigabanya icyo gihembo numubare wibimenyetso, utanga amanota kuri buri kimenyetso aho. Igisubizo nicyitegererezo kiguma kigufi kandi ku ngingo aho guhinduranya ibisubizo kumikino intego.
Ubushishozi
Igihembo cya DPO ni log-ratio hagati ya politiki yateguwe kandi yerekanwe, yegeranijwe kuri buri kimenyetso mubisubizo. Kuberako buri kimenyetso kongeramo irindi jambo (mubisanzwe ryiza), umunzani wigihembo cyigihe hamwe nuburebure bwikurikiranya, kubogama muburyo bwiza bwo kurangiza. SimPO ita urugero rwerekana kandi ikoresha impuzandengo ya log-probabilite kuri buri kimenyetso nkigihembo, hiyongereyeho intego yigihembo. Kugabanya uburebure bikuraho uburebure bwubukanishi, bityo gradients zikunda zigaragaza ubuziranenge kuruta kubara ijambo.
Ingaruka z'Ingamba
Ibyemezo bisobanutse
Iragufasha gutandukanya ibyifuzo bya tekiniki bisobanutse nururimi rwo kwamamaza.
Igiciro na bije
Urashobora kubaza ibibazo byiza byo gushyira mubikorwa mbere yo gukoresha amafaranga cyangwa igihe.
Itsinda hamwe nakazi
Amakipe asangiye ibitekerezo akora ibicuruzwa byiza, politiki, nibyemezo byo kwiga.
Ejo hazaza h'uburebure busanzwe muburyo bwo guhitamo neza
Tegereza kugenzura uburebure kugirango ube ipfundo risanzwe aho kuba igitekerezo. Abashakashatsi bahuza uburebure busanzwe nibihano birebire, ibihembo byateganijwe, hamwe na suite yo gusuzuma ifata uburebure burigihe kugirango bapime ibyiza nyabyo. Mugihe ibihembo byigihembo bigenda neza mugushakisha kubogama, imiyoboro ihuza irashobora gutanga raporo yerekana intsinzi yatsindiye kubusa, kandi abayikoresha bazagenzura neza uburyo terse cyangwa ibisobanuro byikitegererezo bigomba kuba.
Gushyira mu bikorwa Isi
Guhuza umukiriya-wunganira umufasha hamwe na SimPO kuburyo itanga ibisobanuro, ibisubizo nyabyo aho kuba paragarafu zisa neza neza.
Gutangaza 'uburebure bugenzurwa nubutsinzi' kuri AlpacaEval 2 kugirango werekane icyitegererezo cyateye imbere mubyukuri aho kubona ikiganiro gusa.
Ongeraho uburebure busanzwe muri DPO mugihe uhuza neza moderi ya coding kugirango igarure uduce duto duto twukuri, ntabwo tubyibushye.
Gupima igihembo cyigihembo gitanga amanota maremare hejuru, hanyuma ukayatesha agaciro mbere yo kuyakoresha kugirango uhuze umufasha wandika.
Ingaruka & Kurinda
Amakipe atandukanye arashobora gukoresha ijambo rimwe muburyo butandukanye, sobanura intera hakiri kare.
Ibipimo birashobora kugaragara bikomeye mugihe imikorere-yisi-itaringaniye.
Kwirengagiza ubuziranenge bwamakuru na gahunda yo gusuzuma akenshi bitanga ibisubizo byoroshye.
Igishushanyo mbonera
Tangira nururimi rusobanutse rwibisubizo ukeneye.
Toranya intsinzi imwe hamwe nuburyo bumwe bwo gutsindwa mbere yo kwipimisha.
Koresha umuderevu muto hamwe namakuru ahagarariye, ntabwo ari demo yashizweho.
Inyandiko aho Uburebure busanzwe muburyo bwo guhitamo Optimisiyasi ifasha kandi nuburyo bworoshye bworoshye.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Length Normalization in Preference Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Ikigereranyo cya Odds Ibyifuzo byo guhitamo
Ibibazo bikunze kubazwa
What is Length Normalization in Preference Optimization?
Uburebure busanzwe buhindura intego-yoguhuza intego kugirango moderi ihagarike gutsindira kwemererwa kwandika ibisubizo birebire. Nibyiza kuko ibimenyetso byigihembo bidakosowe bisunika ibiganiro kuri verose, ibisubizo bya padi aho kuba byiza rwose.
Ni iyihe myitwarire idakenewe uburebure busanzwe muri DPO bugamije gukumira?
Igihembo cya DPO kidasubirwaho gikura hamwe no kubara ibimenyetso, bityo nta moderi isanzwe yiga ko kuba inshinga nyinshi bikunda gutsinda kugereranya.
Ni ukubera iki ibihembo bisanzwe bya DPO bikunda kwiyongera hamwe nuburebure bwibisubizo?
Ibihembo byuzuye bitondekanya ibipimo-byerekana ibipimo kuri buri kimenyetso, bityo ibimenyetso byinshi mubisanzwe bisobanura ibihembo byinshi.
Nigute SimPO ikemura uburebure bubogamye?
SimPO itanga ibisubizo ukurikije impuzandengo yabo (uburebure-busanzwe) log-bishoboka kandi ikongeramo intego yigihembo, ikuraho inyungu z'uburebure.
Ni ubuhe buryo bwo gusuzuma bufasha kwemeza icyitegererezo cyateye imbere mu bwiza kuruta uburebure gusa?
Uburebure bugenzurwa nubutsinzi (nkuko biri kuri AlpacaEval 2) bihindura kuburebure bwibisubizo bityo inyungu zigaragaza ubuziranenge, ntabwo ari amagambo.