Haɓaka fifikon fifikon ƙima
Haɓaka Zaɓin Ƙarƙashin Ƙarƙashin Ƙarya (ORPO) hanya ce mai kyau mai daidaitawa wacce ke koyar da ƙirar harshe kyakkyawan ɗabi'a da abubuwan da ɗan adam ke so a cikin fasinjan horo ɗaya.
Dubawa
It matters because it skips the usual separate reward model and reference model, making alignment cheaper and simpler.
Zurfafa nutsewa
ORPO, wanda Hong, Lee, da Thorne suka gabatar a cikin 2024, yana haɗa ingantaccen daidaitawa da daidaita fifiko zuwa mataki ɗaya. Yawancin bututun daidaitawa na farko suna yin SFT akan misalai masu kyau, sannan suna gudanar da hanya ta biyu kamar RLHF ko DPO waɗanda ke buƙatar daskararre kwafin ƙirar (wani tunani) da nau'ikan zaɓin zaɓi da aka adana. ORPO yana cire samfurin tunani gaba ɗaya. Asararta tana ƙara wa'adin hukunci ga daidaitaccen maƙasudin alama na gaba: yana ɗaga ƙima da ƙirar ke ba da amsa ga zaɓin da aka zaɓa (wanda aka fi so) yayin turawa ƙasa da rashin daidaito na wanda aka ƙi. Saboda yana amfani da rabon rashin daidaituwa maimakon tazarar yuwuwar log mai ƙarfi, hukuncin yana da laushi, don haka ƙirar ta koyi fifita amsoshi masu kyau ba tare da mantawa da tsararrun tsararraki ba.
Fahimtar Fasaha
Asarar ORPO ita ce asarar giciye-entropy ta SFT tare da log-sigmoid mai nauyi na rabon rashin daidaito tsakanin zaɓaɓɓu da martanin da aka ƙi. Rashin daidaituwa daidai p / (1-p), don haka rabo ya kwatanta yadda mafi kusantar samfurin ya sami amsa mai kyau da mara kyau. Yin amfani da rashin daidaituwa maimakon yuwuwar ɗanyen yana riƙe da ɗanɗano mai ɗanɗano, wanda ke hana wuce gona da iri na alamun da aka ƙi wanda zai iya ƙasƙantar da samfurin da ba a magana ba.
Dabarun Tasiri
Gudu da sikelin
Gudun aikin harshe na iya tafiya da sauri ba tare da sadaukar da daidaito ba.
Shiga ku isa
Yana faɗaɗa damar shiga cikin harsuna da salon sadarwa.
Shawarwari masu haske
Ƙungiyoyi za su iya ciyar da ƙarin lokaci akan hukunci yayin da aiki da kai ke sarrafa maimaitawa.
Makomar Haɓaka Matsayin Rasuwar rashin daidaito
ORPO yana samun karɓuwa saboda yana yanke ƙwaƙwalwar ajiya da ƙididdigewa ta hanyar sauke samfurin tunani, wanda ke da kyau ga ƙungiyoyin daidaitawa akan ƙayyadaddun kayan aiki. Yi tsammanin zai bayyana sau da yawa a cikin buɗaɗɗen girke-girke kuma azaman zaɓi na tsoho a cikin ɗakunan karatu kamar Hugging Face TRL. Aikin gaba zai iya daidaita ma'aunin lambda ta atomatik, haɗe ORPO tare da wasu maƙasudai marasa amfani, kuma a miƙe shi zuwa multimodal da manyan samfura inda riƙe kwafi biyu a ƙwaƙwalwar ajiya yana da tsada.
Aiwatar da Gaskiyar Duniya
Kyakkyawan daidaita samfurin taɗi na 7B mai buɗewa akan nau'ikan zaɓin zaɓi ba tare da loda kwafin tunani na biyu ba, rage ƙwaƙwalwar GPU
Farawa da ke daidaita mataimaki na goyon bayan abokin ciniki don fifita ladabi, amsoshi kan manufofi a cikin horo guda ɗaya maimakon SFT-sai-DPO.
Masu bincike suna kwatanta ORPO da DPO akan saitin bayanai iri ɗaya don nuna kwatankwacin daidaitawa tare da ƙananan ƙididdigewa.
Daidaita ƙirar tushe zuwa yanki na musamman (misali, tsarar doka) inda akwai nau'i-nau'i masu kyau da mara kyau amma kasafin ƙirar lada ba
Hatsari & Tsare-tsare
Abubuwan da aka ruɗe suna iya shigar da rahotanni cikin nutsuwa, kwararar tallafi, ko abubuwan bincike.
Hankali na gaggawa na iya ƙirƙirar sakamako mara daidaituwa a cikin buƙatun iri ɗaya.
Za a iya fallasa bayanan rubutu mai ma'ana idan ikon samun dama yana da rauni.
Taswirar Hanya
Ƙayyade tsarin fitarwa, sautin, da ma'auni masu inganci kafin fitowa.
Amsa a ƙasa tare da amintattun tushe a duk lokacin da daidaito ya shafi mahimmanci.
Ajiye wurin binciken ɗan adam don abubuwan da ake samu masu girma.
Bibiyar tsarin gazawar kuma sake horar da tsokaci ko tafiyar aiki akai-akai.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Odds Ratio Preference Optimization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Haɓaka fifiko kai tsaye
Tambayoyin da ake yawan yi
What is Odds Ratio Preference Optimization?
Haɓaka Zaɓin Ƙarƙashin Ƙarƙashin Ƙarya (ORPO) hanya ce mai kyau mai daidaitawa wacce ke koyar da ƙirar harshe kyakkyawan ɗabi'a da abubuwan da ɗan adam ke so a cikin fasinjan horo ɗaya. Yana da mahimmanci saboda ya tsallake samfurin lada na daban na yau da kullun da samfurin tunani, yana sa daidaitawa ya fi arha da sauƙi.
Menene babban fa'idar ORPO akan hanyoyin kamar DPO?
ORPO yana ninka fifikon koyo cikin asarar SFT kuma baya buƙatar daskararre kwafin ƙirar, adana ƙwaƙwalwar ajiya da ƙididdigewa.
Menene 'rashin rashin daidaituwa' a cikin ORPO ya kwatanta?
ORPO yana amfani da rabon rashin daidaituwa (p/(1-p)) ƙirar tana ba da amsar da aka fi so tare da wanda ba a so.
ORPO yana yin waɗanne ayyuka biyu ne a cikin horo guda ɗaya?
ORPO ya haɗu daidaitaccen maƙasudin alamar alama ta SFT tare da hukuncin fifiko a cikin hasara ɗaya.
Me yasa ORPO ke amfani da rashin daidaito maimakon ɗanyen yuwuwar yuwuwar log don hukunci?
Hukuncin tushen rashin daidaito ya fi sauƙi, yana taimakawa ƙirar da ba a magana ba ta ci gaba da ƙwarewa yayin da har yanzu tana son amsoshi masu kyau.
An fi siffanta asarar ORPO da wace haɗe?
ORPO yana ƙara ma'auni mai ma'aunin log-sigmoid-rashi-rashin ƙima a saman asarar giciye-entropy da ake kulawa.