I-Odds Ratio Preference Optimization
I-Odds Ratio Preference Optimization (i-ORPO) iyindlela yokuhlela kahle efundisa imodeli yolimi ukuziphatha okuhle nokuthandwayo komuntu ekuphumeleleni okukodwa kokuqeqeshwa.
Uhlolojikelele
Ibalulekile ngoba yeqa imodeli yomvuzo ehlukile evamile nemodeli yereferensi, okwenza ukuqondanisa kushibhe futhi kube lula.
I-Deep Dive
I-ORPO, eyethulwe ngu-Hong, u-Lee, no-Thorne ngo-2024, ihlanganisa ukulungisa okugadiwe nokuqondanisa okuthandwayo kube isinyathelo esisodwa. Amapayipi amaningi okuqondanisa aqale enze i-SFT ezibonelweni ezinhle, bese esebenzisa indlela yesibili efana ne-RLHF noma i-DPO edinga ikhophi efriziwe yemodeli (inkomba) kanye namapheya athandwayo agciniwe. I-ORPO isusa imodeli yesithenjwa ngokuphelele. Ukulahlekelwa kwayo kungeza isikhathi sokujeziswa enjongweni evamile yethokheni elandelayo: kuphakamisa izingqinamba imodeli eyabela impendulo ekhethiwe (ekhethwayo) kuyilapho yehlisa amathuba aleyo enqatshiwe. Ngenxa yokuthi isebenzisa isilinganiso sezinkinga kunegebe eliqinile lamathuba elogi, inhlawulo ithambile, ngakho imodeli ifunda ukuthanda izimpendulo ezinhle ngaphandle kokukhohlwa ngokuyinhlekelele isizukulwane esikhuluma kahle.
I-Technical Insight
Ukulahlekelwa kwe-ORPO ukulahlekelwa kwe-SFT cross-entropy kanye ne-log-sigmoid enesisindo yesilinganiso samaphutha elogi phakathi kwezimpendulo ezikhethiwe nezinqatshiwe. Odds alingana no-p/(1-p), ngakho isilinganiso siqhathanisa ukuthi maningi kangakanani amathuba okuthi imodeli ithole impendulo enhle uma iqhathaniswa nembi. Ukusebenzisa izingqinamba esikhundleni samathuba angavuthiwe kugcina ukugqama kuthambile, okuvimbela ukucindezelwa ngokweqile kwamathokheni anqatshiwe angehlisa isithunzi imodeli engabhekiselwe kuyo.
I-Strategic Impact
Isivinini nesikali
Ukugeleza komsebenzi wolimi kungahamba ngokushesha ngaphandle kokudela ukuvumelana.
Finyelela futhi ufinyelele
Yandisa ukufinyelela kuzo zonke izilimi nezitayela zokuxhumana.
Izinqumo ezicacile
Amaqembu angachitha isikhathi esiningi ekwahluleleni kuyilapho i-automation isingatha impinda.
Ikusasa le-Odds Ratio Preference Optimization
I-ORPO izuza amandla ngoba inqamula inkumbulo futhi ibale ngokulahla imodeli yesithenjwa, ekhanga amaqembu ahlela kahle ihadiwe elinganiselwe. Lindela ukuthi ivele kaningi kumaresiphi omthombo ovulekile futhi njengenketho ezenzakalelayo kumalabhulali afana ne-Hugging Face TRL. Umsebenzi wesikhathi esizayo cishe uzoshuna isisindo se-lambda ngokuzenzakalela, uhlanganise i-ORPO nezinye izinjongo ezingenazo ireferensi, futhi uyinwebe kumamodeli e-multimodal namamodeli amakhulu kakhulu lapho ukubamba amakhophi amabili ngekhanda kubiza khona.
Ukuqaliswa Komhlaba Wangempela
Ukuhlela kahle imodeli yengxoxo ye-7B yomthombo ovulekile kumapheya athandwayo ngaphandle kokulayisha ikhophi yesibili eyireferensi, ukunciphisa inkumbulo ye-GPU
Isiqalo sokuqondisa umsizi wokwesekwa kwamakhasimende ukuze akhethe izimpendulo ezinesizotha, eziphathelene nenqubomgomo ekugijimeni okukodwa kokuqeqeshwa esikhundleni se-SFT-bese-DPO
Abacwaningi abaqhathanisa i-ORPO ne-DPO kudathasethi efanayo ukuze babonise ukuqondana okuqhathanisekayo nekhompuyutha ephansi
Ukulungisa imodeli eyisisekelo esizindeni esikhethekile (isb., ukubhala okusemthethweni) lapho amapheya ezibonelo ezinhle nezimbi atholakalayo kodwa isabelomali semodeli yomvuzo singekho.
Izingozi & Guardrails
Amaqiniso akhonjiwe angafaka ngokuthula imibiko, ukugeleza kosekelo, noma imiphumela yocwaningo.
Ukuzwela okusheshayo kungadala imiphumela engahambisani kuzo zonke izicelo ezifanayo.
Idatha yombhalo ebucayi ingase idalulwe uma izilawuli zokufinyelela zibuthakathaka.
Ukuqalisa Umhlahlandlela
Chaza ifomethi yokuphumayo, ithoni, namazinga wekhwalithi ngaphambi kokukhishwa.
Izimpendulo eziyisisekelo ngemithombo ethembekile noma nini lapho ukunemba kubalulekile.
Gcina indawo yokuhlola isibuyekezo somuntu ukuze uthole imiphumela ephezulu.
Landela amaphethini okuhluleka futhi uqeqeshe kabusha imiyalo noma ukuhamba komsebenzi njalo.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Odds Ratio Preference Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
Ukuthuthukisa Okuncamelayo Okuqondile
Imibuzo evame ukubuzwa
Iyini i-Odds Ratio Preference Optimization?
I-Odds Ratio Preference Optimization (i-ORPO) iyindlela yokuhlela kahle efundisa imodeli yolimi ukuziphatha okuhle nokuthandwayo komuntu ekuphumeleleni okukodwa kokuqeqeshwa. Ibalulekile ngoba yeqa imodeli yomvuzo ehlukile evamile nemodeli yereferensi, okwenza ukuqondanisa kushibhe futhi kube lula.
Iyiphi inzuzo enkulu ebonakalayo ye-ORPO kunezindlela ezifana ne-DPO?
I-ORPO igoqa ukufunda okuthandwayo ekulahlekeni kwe-SFT futhi ayidingi ikhophi yesithenjwa efriziwe yemodeli, ilondoloza inkumbulo nokubala.
Iqhathanisa 'isilinganiso sezinkinga' ku-ORPO?
I-ORPO isebenzisa isilinganiso sama-odds (p/(1-p)) imodeli yabela impendulo encanyelwayo uma iqhathaniswa nengakhethwanga.
I-ORPO yenza imiphi imisebenzi emibili ekuqeqesheni okukodwa?
I-ORPO ihlanganisa inhloso yethokheni elandelayo ye-SFT nenhlawulo ethandwayo ekulahlekelweni okukodwa okukodwa.
Kungani i-ORPO isebenzisa izingqinamba kunomehluko ongahluziwe wamathuba elogi ukuze uthole inhlawulo?
Inhlawulo esekelwe kumathuba ithambile, isiza imodeli engabhekiselwe kuyo ukuthi igcine isizukulwane esiqephuzayo ngenkathi ikhetha izimpendulo ezinhle.
Ukulahlekelwa kwe-ORPO kuchazwa kangcono ngokuthi iyiphi inhlanganisela?
I-ORPO yengeza ithemu elinesisindo se-log-sigmoid odds-ratio phezu kokulahlekelwa kwe-cross-entropy egadiwe.