Odds Ratio Preference Optimization
Odds Ratio Preference Optimization (ORPO) inzira yekumisikidza zvakanaka inodzidzisa mhando yemutauro maitiro akanaka uye zvido zvevanhu mukupasa kumwe chete.
Pfupiso
It matters because it skips the usual separate reward model and reference model, making alignment cheaper and simpler.
Kudzika Kwakadzika
ORPO, yakaunzwa naHong, Lee, uye Thorne muna 2024, inosanganisa yakatariswa kugadzirisa uye kurongeka kwekuda kuita nhanho imwe. Mapaipi akawanda ekuenzanisa anotanga aita SFT pamienzaniso yakanaka, wozomhanyisa nzira yechipiri seRLHF kana DPO inoda kopi yakaomeswa nechando yemuenzaniso (referensi) pamwe neakachengetwa mapeya ekuda. ORPO inobvisa iyo referensi modhi zvachose. Kurasikirwa kwayo kunowedzera nguva yechirango kune yakajairwa-inotevera-chiratidzo chinangwa: inosimudza kusawirirana iyo modhi inopa kune yakasarudzwa (inofarirwa) mhinduro uku ichisundira pasi kusawirirana kweyakarambwa. Nekuti inoshandisa iyo odds reshiyo pane yakasimba log-mukana gap, chirango chakapfava, saka modhi inodzidza kufarira mhinduro dzakanaka pasina njodzi yekukanganwa chizvarwa chinotsetseka.
Technical Insight
Kurasika kweORPO ndiko kurasikirwa kweSFT muchinjiko-entropy pamwe nekuremerwa kwelogi-sigmoid yereshiyo yelogi odds pakati pemhinduro dzakasarudzwa uye dzakarambwa. Odds yakaenzana p/(1-p), saka reshiyo inoenzanisa kuti modhi yacho ingangowana mhinduro yakanaka sei maringe neiyo yakaipa. Kushandisa zvipingamupinyi pachinzvimbo chemukana wakasvibira kunochengeta musiyano wakapfava, izvo zvinodzivirira kudzvinyirira zvakanyanya kwezviratidzo zvakarambwa zvinogona kudzikisira modhi isingatariswe.
Strategic Impact
Kumhanya uye chiyero
Mutauro workflows inogona kufamba nekukurumidza pasina kupira kuenderana.
Svika uye svika
Inopamhidzira kupinda mumitauro yese nemataera ekutaurirana.
Sarudzo dzakajeka
Zvikwata zvinogona kupedza nguva yakawanda pakutonga uku otomatiki ichibata kudzokorora.
Ramangwana reOdds Ratio Preference Optimization
ORPO iri kuwana traction nekuti inocheka ndangariro uye komputa nekudonhedza referensi modhi, iyo inokwezva zvikwata kunyatso-tung pahardware shoma. Tarisira kuti ionekwe kazhinji muakavhurika-sosi yekubika uye senge yakasarudzika sarudzo mumaraibhurari seHugging Face TRL. Basa remangwana rinogona kurongedza uremu hwe lambda otomatiki, kusanganisa ORPO nezvimwe zvinangwa-zvisina referensi, uye kuiwedzera kune multimodal uye yakakura kwazvo modhi uko kubata makopi maviri mundangariro kunodhura.
Real-World Implementation
Kunyatsogadzirisa yakavhurika-sosi 7B chat modhi pane zvaunofarira vaviri pasina kurodha yechipiri referensi kopi, nepakati GPU memory.
Kutanga kuenzanisa mutengi-rutsigiro mubatsiri kuti asarudze zvine hunhu, pane-policy mhinduro mune imwe dzidziso inomhanya panzvimbo yeSFT-ipapo-DPO.
Vatsvagiri vanofananidza ORPO vachipesana neDPO pane imwecheteyo dataset kuratidza kuenzanirana kuenderana neyakaderera compute
Kuchinjisa modhi yehwaro kune yakasarudzika dura (semuenzaniso, kunyora zviri pamutemo) uko yakanaka uye yakaipa mienzaniso miviri iripo asi mubairo-modhiyo bhajeti haisi.
Njodzi & Guardrails
Chokwadi chehuroyi chinogona kupinda chinyararire mishumo, kuyerera kwetsigiro, kana tsvakiridzo.
Kunzwa nekukasira kunogona kugadzira mhedzisiro isingaenderane pane zvikumbiro zvakafanana.
Sensitive text data inogona kuburitswa kana zvidhiraivho zvisina kusimba.
Implementation Roadmap
Tsanangura chimiro chekubuda, toni, uye mhando zviyero usati waburitsa.
Mhinduro dzepasi neakavimbika masosi pese pazvine basa.
Chengetedza ongororo yekuongorora yemunhu kune yakakwira-stake zvinobuda.
Tevera maitiro ekutadza uye dzidzisazve kukurudzira kana mafambiro ebasa nguva nenguva.
Ramba Uchiongorora
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Odds Ratio Preference Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gaidhi rinotevera
Direct Preference Optimization
Mibvunzo inowanzo bvunzwa
What is Odds Ratio Preference Optimization?
Odds Ratio Preference Optimization (ORPO) inzira yekumisikidza zvakanaka inodzidzisa mhando yemutauro maitiro akanaka uye zvido zvevanhu mukupasa kumwe chete. Izvo zvine basa nekuti inosvetuka yakajairwa yakapatsanurwa modhi uye referenzi modhi, zvichiita kuti kurongeka kudhure uye kuve nyore.
Ndeipi yakakosha mukana weORPO pamusoro penzira dzakaita seDPO?
ORPO inopeta kudzidza kwekuda mukurasika kweSFT uye haidi kopi yakaomeswa yereferensi yemuenzaniso, kuchengetedza ndangariro uye komputa.
Chii chinofananidzwa neiyo 'odds ratio' muORPO?
ORPO inoshandisa reshiyo yemaodds (p/(1-p)) modhi yacho inopa mhinduro yakasarudzika maringe neyakasadiwa.
ORPO inoita mabasa api maviri mukupasa kumwe chete?
ORPO inosanganisa yakajairwa SFT inotevera-chiratidzo chinangwa nechirango chekuda mukurasikirwa kumwe kwakabatana.
Sei ORPO ichishandisa zvipingamupinyi pane mutsauko wegigi-mukana wechirango?
Iwo maodd-based chirango chakapfava, zvichibatsira iyo isina kurongeka modhi kuchengeta chizvarwa chakatsetseka uchiri kuda mhinduro dzakanaka.
Iko kurasikirwa kweORPO kunotsanangurwa zvakanyanya sekuti ndeipi musanganiswa?
ORPO inowedzera inorema log-sigmoid odds-ratio temu pamusoro pekutariswa kwemuchinjiko-entropy kurasikirwa.