Ikigereranyo cya Odds Ibyifuzo byo guhitamo
Igipimo cyiza cya Optimisiyoneri (ORPO) nuburyo bwiza bwo guhuza neza bwigisha icyitegererezo cyururimi imyitwarire myiza nibyifuzo byabantu mumwanya umwe wamahugurwa.
Incamake
It matters because it skips the usual separate reward model and reference model, making alignment cheaper and simpler.
Kwibira cyane
ORPO, yatangijwe na Hong, Lee, na Thorne mu 2024, ikomatanya kugenzura neza no guhuza ibyifuzo mu ntambwe imwe. Imiyoboro myinshi ihuza ibanza ikora SFT kurugero rwiza, hanyuma ikore uburyo bwa kabiri nka RLHF cyangwa DPO bisaba kopi yahagaritswe yicyitegererezo (reference) hiyongereyeho kubikwa kubiri. ORPO ikuraho moderi yerekana rwose. Igihombo cyayo kongeramo igihano kubisanzwe bikurikira-ikimenyetso: bizamura impinduka icyitegererezo giha igisubizo cyatoranijwe (gikunzwe) mugihe gisunika ibitagenda neza. Kuberako ikoresha ikigereranyo kidasanzwe aho kuba ikinyuranyo gikomeye-gishoboka, igihano kiritonda, nuko rero icyitegererezo cyiga gutanga ibisubizo byiza utibagiwe nibisekuru neza.
Ubushishozi
Igihombo cya ORPO nigihombo cya SFT cross-entropy hiyongereyeho log-sigmoid iremereye yikigereranyo cyibiti bitandukanya ibisubizo byatoranijwe kandi byanze. Impanuka zingana p / (1-p), igipimo rero kigereranya uburyo bishoboka cyane ko icyitegererezo kibona igisubizo cyiza nikibi. Gukoresha ibidashoboka aho kuba bibisi byoroshye bituma itandukaniro ryoroheje, ririnda gukabya gukabya ibimenyetso byangwa bishobora gutesha agaciro icyitegererezo.
Ingaruka z'Ingamba
Umuvuduko n'igipimo
Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
Kugera no kugera
Yagura uburyo bwindimi nuburyo bwo gutumanaho.
Ibyemezo bisobanutse
Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.
Igihe kizaza cya Odds Ikigereranyo cyo guhitamo
ORPO irimo kwiyongera kuko igabanya kwibuka no kubara mugushakisha icyitegererezo, gikurura amakipe gutunganya neza ibyuma bike. Witege ko izagaragara cyane mubisubizo bifungura-isoko kandi nkuburyo busanzwe mumasomero nka Hugging Face TRL. Akazi kazoza karashobora guhuza uburemere bwa labda mu buryo bwikora, guhuza ORPO nizindi ntego zidafite aho zihurira, no kuyigeza kuri moderi nyinshi kandi nini cyane aho gufata kopi ebyiri murwibutso bihenze.
Gushyira mu bikorwa Isi
Kuringaniza neza-isoko ya 7B yo kuganira kubintu byombi utabanje gupakira kopi ya kabiri yerekana, kugabanya ububiko bwa GPU
Gutangira guhuza umufasha-wunganira umufasha kugirango ahitemo ikinyabupfura, kuri politiki ibisubizo mumahugurwa amwe akora aho kuba SFT-hanyuma-DPO
Abashakashatsi bagereranya ORPO na DPO kuri dataset imwe kugirango berekane guhuza kugereranya na compte yo hasi
Guhuza icyitegererezo cyibanze kumurongo wihariye (urugero, gutegura amategeko) aho ingero nziza kandi mbi zombi ziboneka ariko ingengo yimishinga ntangarugero ntabwo
Ingaruka & Kurinda
Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.
Igishushanyo mbonera
Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
Komeza kugenzura abantu kugenzura ibisubizo byinshi.
Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Odds Ratio Preference Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Guhitamo neza
Ibibazo bikunze kubazwa
What is Odds Ratio Preference Optimization?
Igipimo cyiza cya Optimisiyoneri (ORPO) nuburyo bwiza bwo guhuza neza bwigisha icyitegererezo cyururimi imyitwarire myiza nibyifuzo byabantu mumwanya umwe wamahugurwa. Nibyingenzi kuko isimbuka bisanzwe bisanzwe ibihembo byigihembo nicyitegererezo, bigatuma guhuza bihendutse kandi byoroshye.
Ni izihe nyungu nyamukuru za ORPO kurenza uburyo nka DPO?
ORPO ikubye ibyifuzo byo kwiga mugihombo cya SFT kandi ntikeneye kopi yerekana ifunze yikitegererezo, ikiza ububiko na compte.
Ni iki 'ikigereranyo kidasanzwe' muri ORPO kigereranya?
ORPO ikoresha ikigereranyo cyibidasanzwe (p / (1-p)) icyitegererezo giha igisubizo cyatoranijwe nicyatanzwe.
ORPO ikora imirimo ibiri murwego rumwe rwamahugurwa?
ORPO ikomatanya bisanzwe SFT ikurikira-token intego hamwe nigihano cyo guhitamo mugihombo kimwe.
Ni ukubera iki ORPO ikoresha ibidasanzwe aho gukoresha ibiti bito-bishoboka kubihano?
Igihano gishingiye ku gihano kiroroshye, gifasha icyitegererezo kidasubirwaho gukomeza ibisekuruza neza mugihe ugihitamo ibisubizo byiza.
Igihombo cya ORPO cyasobanuwe neza nkubuhe?
ORPO yongeramo uburemere bwa log-sigmoid idasanzwe-igereranyo hejuru yigihombo cyagenzuwe na cross-entropy.